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Your performance hints, please ... 

This book will be updated as more information becomes available. 
You can submit performance hints for possible publication in this 
book. Use the reader's comment or Performance Notebook input 
form(s) at the back of this book or send your information to: 

IBM Corporation 
Publications Department 
Department D58, Building 706-2 
PO Box 390 

Poughkeepsie, New York 12602 
ATTN: Performance Notebook 

When submitting performance hints, see page iii for details. 
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The recommendations given in this manual are based on experience with 0S/VS2 MVS 
(Multiple Virtual Storage - VS2 Release 3.7 with SUs 5 and 7 applied, and any subsequent 
VS2 releases) at internal IBM installations, at field test installations, and at user installations. 
As such, this material has not been submitted to any formal IBM test. Potential users should 
evaluate the applicability of the recommendations in their environment before implementation 
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What To Expect From This Book 



The subject of this book is performance evaluation: the process of tuning your 
system to meet your performance expectations and to optimize use of the system 
resources. There is no "cookbook" method to do a performance evaluation that 
will automatically provide you with specific performance solutions to apply to your 
system; the responsibility of identifying performance improvements for your 
system rests with you, the user. As a result, do not expect to find sure-fire 
performance solutions outlined in this book; there are no easy answers. Rather, 
this book attempts to provide necessary information foi you to evaluate the 
performance of your system in a disciplined way, with some degree of confidence 
that the evaluation will succeed in identifying ^'owr system's problem areas and 
specific solutions for your particular installation. 

The information is based on the experience of many MVS performance analysts. 
It is organized into the following sections: 

• Introduction, which identifies the significant problems that have frequently 
hampered performance evaluation and outlines the steps necessary to approach 
performance evaluation in a disciplined way. 

• Part I, Planning and Preparing for a Performance Evaluation, which includes 
three chapters; 

- Chapter I.l, "Defining Performance Objectives," which describes factors you 
should include in defining what you expect from the system- 

- Chapter 1.2, "Selecting Measurement Tools," which provides an overview of 
tools available from IBM and describes other sources of MVS data. 

- Chapter 1.3, "Pre-initialization MVS Performance Factors," which outlines 
basic performance factors applicable to all system control programs, but 
which were often ignored in early experience with MVS systems. 

• Part II, Performance Analysis, which documents a methodology for identifying 
potential problem areas in your system. The intent of this part is to help you 
focus on the area of most probable significant performance improvement in 
your system; the methodology is a combination of performance theory and 
MVS tuning experience. This part includes the following chapters : 

- Chapter II. 1, "Steps in a Performance Problem Analysis" 

- Chapter 11.2, "Investigating the Use of Processor Time" 

- Chapter II.3, "Investigating the Use of I/O Resources" 

- Chapter II.4, "Investigating the Use of Real Storage" 

- Chapter 11.5, "User-oriented Performance Problem Analysis" 
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Chapters 11.2-11.5 consist of background information that describes the general 
approach for investigating the resource or type of problem documented in the 
chapter, followed by bulletins that document specific areas to investigate and 
possible solutions to bottlenecks detected in those areas. The bulletins are 
numbered in the format x.y.O, where x represents the number of the chapter 
within Part II; y represents the number of the bulletin within the chapter; 
and allows for the addition of bulletins in future updates to this book. 

Note that chapter 11.5 describes potential causes of delay that are also 
documented in the chapters on specific resources (for example, access to the 
processor; I/O delays; paging and swapping). The focus of chapter II. 5, how- 
ever, differs from the focus in chapters II.2-II.4. In chapter II.5, you are 
examining each resource in Ught of specific work that is not meeting its objective; 
in chapters II.2-IL4, you are examining total usage of the resource. 

• Part III, Performance Hints, which documents specific performance suggestions 
and considerations. This section is organized alphabetically by the topic of the 
hint. The choice of the title of each topic is somewhat arbitrary, in that many 
of the topics are related (for example, domain control hints and demand paging 
hints could also be considered SRM or real storage hints). We have tried to be 
as specific as possible in titling each topic and to avoid broad, general topics. 
The title page of Part III and the table of contents list the titles chosen for the 
topics. In selecting hints to apply to your system, be sure they address a 
problem area identified in your system. The analysis documented in Part II 
should help you to identify problem areas. 

The information in the book is based on a Release 3.7 MVS system with SUs 5 
and 7 applied, unless otherwise indicated. 

Although configuration planning and capacity planning are related to 
performance evaluation, they are not included in this book. They are addressed 
implicitly only to the extent that the analysis might help you to identify errors in 
configuration planning or the point at which additional capacity is, needed. This 
book also does not address benchmarking aspects of performance evaluation. 

It is assumed that the user of this book is responsible for the performance 
evaluation of the installation's system; he or she should have experience in tuning 
complex systems and have a thorough knowledge of MVS concepts and facilities. 
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Submitting Performance Hints 



This book will be updated as additional information becomes available. You can 
submit performance hints for possible publication in this book. Use the reader's 
comment or Performance Notebook input form(s) at the back of this book or send 
your information to: 

IBM Corporation 
Publications Department 
Department D58, Building 706-2 
PO Box 390 

Poughkeepsie, New York 12602 
ATTN: Performance Notebook 

It is understood that IBM and its affiUated companies shall have the nonexclusive 
right, in their discretion, to use, copy, and distribute all submitted information or 
material, in any form, for any and all purposes, without any obligation to the 
submitter, and that the submitter has the unqualified right to submit such informa- 
tion upon such basis. 

When submitting performance hints, indicate the system and release level of 
your system and the SUs installed on your system. 



Associated Publications and Selectable Units 

The pubHcations listed below are referenced in the text of this book. References 
in the book usually use an abbreviated form of the title and do not include the 
order number. 

• 0SIVS2 Conversion Notebook, GC28-0689 

• OS/VS Linkage Editor and Loader, GC26-3 8 1 3 

• 0SIVS2 MVS System Programming Library: JES2, GC23-0001 

• OS/VS System Programming Library: Initialization and Tuning Guide, 
GC28-0681 

• 0S/VS2 TCAM User's Guide, GC30-2045 

• 0S/VS2 TCAM Logic, SY30-2040 

The preceding Ust does not include publications describing measurement tools 
referenced in this book. Figure 1.4, "General Information on MVS Measurement 
Tools," lists the order numbers of documentation on measurement tools available 
from IBM. 



Selectable units (SUs) are referred to by their SU number. The following list 
gives the full names and ID numbers of SUs mentioned in this book: 

SU 1 - VTAM Level 2, VS2.03.801 

SU 2 - TC AM Level 9 , VS2 .03 .802 

SU4 - Scheduler Improvements, VS2.03. 804 

SUS - Supervisor Performance #1,VS2. 03. 805 

SU6 - Attached Processor System, VS2. 03 .806 

SU7 - Supervisor Performance #2, VS2.03 .807 

SU8 - Data Management, VS2. 03. 808 

SU 10 - IBM 3800 Printing Subsystem, VS2.03.810 

SU15 - SMP Enhancements, VS2.03.8 15 
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Introduction 



As observed in several existing MVS systems, problems such as the following 
frequently hamper performance evaluations: 

• Users have not always clearly defined what they expect from the system. 
Unsatisfactory performance is described vaguely and tuning efforts, which 
should be based on a specific problem description, are not. 

• Installations often apply "corrective" actions to the system based on the 
reputation of those actions rather than in response to a problem area identified 
in the system. 

• Faced with a morass of measurements of the system, users often don't know 
which measurements to focus on or how to judge the reported values of the 
measurements they do focus on. 

Such problems can be minimized by approaching the performance evaluation in 
a disciplined way. 

A disciplined performance evaluation should include the following steps: 

1 . Define the work the system is expected to do - that is, begin by setting 
performance objectives for your system. Chapter I.l describes the importance 
of, and factors to be included in, the definition of performance objectives. 

2. Identify and, if necessary, obtain or develop required measurement tools. 
The goal of the performance evaluation is to make the most effective use of 
the system resources in order to meet the installation's performance objectives. 
Since measurement tools are the basic means to determine the use of system 
resources — and to determine if you are meeting your objectives — any per- 
formance evaluation is limited by the measurement tools available. Chapter 1.2 
is an overview of several MVS measurement tools and Part II, Performance 
Analysis, describes specific measurements you can use to identify potentially 
critical resources in your system. 

3. Ensure that performance-related factors are addressed when setting up the 
system . These factors include sysgen and initialization parameters that 
have performance implications and basic performance factors that should be 
addressed for any system control program. See chapter 1.3. 

4. Once the system is up and stable (that is, availability and reliability require- 
ments are met), measure against the performance objectives estabHshed in 
step 1. Chapter I.l suggests ways to measure against the specific objectives 
described in that chapter. If objectives are met, skip to step 8. 
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5. If objectives are not met, focus on areas of probable significant performance 
improvement . Part II, Performance Analysis, describes specific measurements 
that can be used to guide the analysis, depending on the type of performance 
problem being experienced. 

6. Having focused on areas of probable improvement, identify and implement 
corrective actions . Part III documents specific performance suggestions. 

7. Evaluate the results of the actions implemented - remeasure against the 
objectives. Often, one bottleneck will hide other bottlenecks m the system. 
If objectives still are not met, return to step 5. 

8. Due to the changing nature of any system, it is important to monitor 
performance once objectives are met. Workload management and I/O resource 
management are key areas that require continual monitoring. By monitoring 
the system and its use of major resources, you can address performance 
problems before they become critical. The topic "Monitoring Performance" 
in chapter 1.2 describes several key measurements that are useful to track. 
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Part I: Planning and Preparing for a Performance Evaluation 



Chapter 1,1: Defining Performance Objectives 

Chapter 1.2: Selecting Measurement Tools 

Chapter 1.3 : Pre-initialization MVS Performance Factors 
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Chapter I.l Defining Performance Objectives 



Note: In this book, the phrase "performance objectives" refers to the specific 
goals and expectations an installation sets for its system. To refer to performance 
objectives specified in the Installation Performance Specification (IPS), "IPS 
performance objective" is used. Do not confuse the IPS performance objective 
with specifying performance objectives for your installation. The IPS is the means 
to achieve the objectives you set for each category of work - the means of 
implementing the resource usage, priorities, and trade-offs of different categories 
of work. Defining discipUned performance objectives, as described in this chapter, 
is important to creating an effective IPS performance objective. 

The definition of performance objectives has two goals: 

• To state what is expected of the system in specific terms for each category of 
work (for example, TSO trivial versus nontrivial transactions) at each distinct 
period of time (for example, prime shifts versus off-shifts and peak periods 
within each shift). 

• To understand and document the resources required to meet the objectives 
defined as the first goal. 

From the nature of these two goals, the definition of performance objectives is 
iterative. You should expect to update your performance objectives as the 
workload changes, as better understanding of resource requirements is gained, as 
resource requirements of the work change, and as turnaround and response time 
requirements change. Detailed performance objectives, however, will make such 
changes noticeable and will help identify solutions to performance problems that 
arise because of the changing nature of the workload. The definition of 
performance objectives is not a trivial task, but it is essential to a disciplined 
performance analysis and for planning new applications and additional work. 

Figure I.l lists the steps in defining performance objectives; subsequent topics 
in this chapter provide additional information on the steps. 
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1 . Define the terms in whicli to specify objectives. 



2. Determine how the objectives will be measured. 

• Note discrepancies between the measurements and what the user sees. 



3. Document the current workload — amount and categories of work. For example: 

• TS6 — trivial and non-trivial transactions 

• batch — job classes 

• IMS — transaction types 

Examine existing workload categories for their effectiveness in distinguishing work 
that requires different priorities and different objectives. 



4. Set objectives for each category of work. 



5. Measure and document resources used by each category of work. 

a. Correct any ineffective classifications of work, based on resources used. 

b. Determine controls to be used to enforce resource usage rules for the different 
categories. 

c. Review reasonableness of the objectives in light of resource requirements and set 
capacity limits for each category of work. 



6. Measure against the objectives. 

a. If measured objectives meet defined objectives, monitor performance. See the 
topic "Monitoring Performance" in Chapter 1.2. 

b. If measured objectives do not meet defined objectives, analyze the system to 
identify problem areas — see Part n. 



Figure 1. 1 Steps in Defining Performance Objectives 



Steps 1 & 2: Selecting and Measuring Objectives 

The first step in defining performance objectives is to choose the terms in which 
you will specify objectives. There are two basic types of performance objectives: 
user-oriented, which reflect the way an end user would rate the services provided 
by the system; and system-oriented, which reflect the workload that must be 
supported on a system level. User-oriented objectives include response time for 
interactive work (TSO, IMS, CICS, VSPC, and so on) and turnaround time for 
batch work. System-oriented objectives include batch throughput, interactive 
transaction rate, and the number of concurrent interactive users. 

The distinction between system-oriented and user-oriented objectives is not 
merely academic. Achieving optimal system-oriented objectives (that is, getting as 
much work through the system as possible) implies achieving the highest utiliza- 
tion of the system resources - processor, real storage, charmels, control units, and 
devices. Achieving optimal user-oriented objectives implies the availabiUty of any 
resource when required. Ensuring the proper workload mix in terms of required 
resources will help avoid conflicts between meeting user-oriented and system- 
oriented objectives. 
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In addition, the initial focus of a tuning effort is dictated by the type of 
objective not being met. When investigating user-oriented performance problems, 
you have to first determine to what resource(s) the work is being denied access 
and try to increase its access to that resource(s). The initial focus is on workload 
management — favoring access of one type of work over other types of work to 
needed resources. When system-oriented problems occur, the focus is on resource 
management — identifying the critical resource(s) in the system and increasing 
the effective utilization of the resource(s). Part II, Performance Analysis, 
describes in detail the approaches for investigating user-oriented and system- 
oriented performance problems. 

When choosing the terms in which to define your objectives, you must also 
determine how the objectives will be measured and reported. For user-oriented 
objectives, you must note any differences between the measured objectives and 
what the user sees. Times reported by measurement tools are usually system 
elapsed times and do not include delays such as job output distribution, polling 
delays at a terminal, and so on. 

The following paragraphs describe TSO response time, TSO transaction rate, 
IMS response time, and batch turnaround time and throughput, including sugges- 
tions on how they can be measured. 



TSO Response Time 

TSO response time is directly related to the time it takes for a TSO transaction to 
complete, as reported by RMF or MF/1 . The time between transaction start and 
transaction end is accumulated for all TSO users residing in a particular 
performance group and is reported on the MF/1 or RMF workload activity 
report as "average time of ended transaction." A TSO transaction, as defined by 
SRM and reported by RMF and MF/1 , is usually signalled by interaction between 
the user and the system; for example: the execution of a single command or 
subcommand; in EDIT input mode, when the user presses the ENTER key. 
Specifically, the start of a TSO transaction is signalled by either of the following: 

• SRM receives a user-ready SYSEVENT for an address space that was swapped 
out because it was in terminal wait. (Note that a terminal-wait SYSEVENT 
does not force an unconditional swap-out of the TSO address space unless 
there is no other ready work in that address space.) 

• SRM receives a TGETTPUT SYSEVENT issued by the TIOC, indicating that a 
TGET has been completed for a particular TSO address space (that is, an 
address space that was not swapped out due to terminal wait). 

The end of a TSO transaction is signalled by either of the following: 

• The TIOC issues a terminal wait SYSEVENT because the TSO address space 
enters input or output wait (IWAIT or OWAIT) and the address space has no 
more ready work. 
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• SRM receives another TGETTPUT SYSEVENT indicating that a subsequent 
command has been received. This signals the end of the previous transaction 
and the beginning of a new transaction. 

Note, however, that the response time as measured by RMF or MF/1 is an average 
and can be different from that which individual end users see. In addition, the 
RMF or MF/1 reported time does not include factors such as line speed, control 
unit delay, polling delay, or the length of time the teleprocessing access method 
requires to decode the user i.d. from the input data stream. Some experiments 
have shown as much as a two-second delay due to such factors. 



TSO Transaction Rate 

TSO transaction rate is the number of TSO transactions that complete per unit of 
time. There is a direct relationship between transaction rate and response time, 
as illustrated in the following formula: 

number of active terminals 
transaction rate = 



user time + response time 

where user time consists of thinking and typing time. As response time decreases, 
transaction rate increases. It has also been observed that faster response time 
often indirectly results in less user time, again resulting in an increased transaction 
rate. Note that if transaction rate, response time, and the number of active 
terminals are known, this formula can be used to compute user time. 

The "ended transactions" field on the RMF or MF/1 workload activity report 
gives the number of transactions that ended in the interval; the transaction rate 
can then be computed as the number of ended transactions divided by the interval. 
Note, however, that, for VTAM, end-of-screen on 3270s is also considered a trans- 
action in the RMF or MF/1 counts and, therefore, is reflected in the "ended 
transactions" field. 



IMS Response Time 

The transaction response report, produced by the IMS/VS statistical analysis 
utility programs, reports response times by transaction type, including: total 
number of responses; longest and shortest responses; and average response times 
achieved by 95%, 75%, 50%, and 25% of the transactions. The time reported is 
from complete receipt of the input message until the response message to the 
terminal is successfully dequeued. Input to the IMS/VS statistical analysis utility 
programs is the IMS log tape. To obtain a transaction response report for certain 
time periods, execute the IMS/VS log transaction analysis utility program with 
appropriate start and end times to obtain a log tape for the desired period. For 
details, see the IMS/VS Utilities Reference Manual. 
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Batch Throughput and Turnaround Time 

Batch throughput is available from the RMF or MF/1 workload activity report. 
For batch work, a transaction as reported by RMF or MF/1 is a job, unless 
different performance group numbers are assigned to the steps of a job ; in such 
cases, each job step is considered a transaction. If performance group numbers are 
assigned only on a job basis, you can compute batch throughput as "ended trans- 
actions" for batch performance groups, divided by the RMF or MF/1 interval. 

Using a data reduction program, you can compute turnaround time and 
throughput from information in SMF record type 5, which contains fields for the 
time the reader recognized the JOB card and the time the job terminated. Note 
that the turnaround time computed via SMF does not include delays entering the 
job into the system and distributing output, which increase turnaround time from 
the user's point of view. 



Step 3: Documenting the Workload 

After deciding on and defining the terms in which to measure performance, the 
next step in writing performance objectives is to understand and document the 
current workload on the system. This includes the amount and the categories 
of work: 

• priority of the work. 

• different periods of time during which objectives and priorities vary for the 
same work. 

• resource requirements of the work. 

• the types of users that require different objectives. 

• the ability to track and report on work according to the installation's needs — 
for example, department breakdowns. 

Start with the categories of work that were defined under the prior system: 
batch job classes, IMS transaction types, and so on. Usually, however, these 
categories were not fully defined. Figure 1.2 suggests data to collect to fully 
define each category of work. (Note that figure 1.2 includes resource require- 
ments for each category of work. When initially defining the categories, the 
resource requirements will probably reflect expected resource usage; measuring 
actual resource usage on MVS is described in more detail in step 5.) 

Once the categories are defined, review them for apparent errors. The purpose 
of the different categories is to distinguish work according to different resource 
requirements, different objectives that must be met, different priorities, and so 
on. For example, all jobs submitted from similar development groups in different 
locations are expected to receive the same turnaround time. However, because of 
distribution of the completed work to different locations, and possible time 
differences in actually returning output to the submitters, you might want to 
further separate this work — to give priority to jobs that have to be distributed to 
locations in different time zones, where delays in turnaround time can have a 
more significant effect on the users. 
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Note that the factors Hsted in figure 1.2 should be determined for each level of 
each category of work. For example, the factors should be determined for batch, 
TSO, and/or IMS or other subsystems. Within each subsystem, the factors should 
be defined for further breakdowns of that type of work: batch work divided into 
job classes; TSO and IMS into types of transactions; all categories divided by 
peak hours and off-shifts. 



Batch 


by job class 


for total batch 


number of jobs per unit of time (e.g., hour, siiift) 

arrival rate 

average elapsed time 

processor time per hour/shift/day 

number of EXCPs 

average service units required to complete 

virtual storage requested 

number of disk mounts 

number of tape mounts 

cards in 

cards out 

lines printed 

scratch space 

contiguous scratch space 


X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 


X 
X 

X 


TSO 


by transaction type 


for total TSO 


minimum/maximum/average number of users 

logged on per hour/shift/day 
average processor time per transaction 
average I/O time per transaction 
average service units required to complete 
TSO region size 


X 
X 
X 


X 
X 
X 
X 
X 


IMS 


by transaction type 


for total IMS 


minimum/maximum/average number of active 

terminals per hour/shift/day 
average number of transactions per second 
average number of transactions per schedule 
control blocks (PSBs and DMBs) loaded per 

schedule 
average number of DB calls per transaction 
average number of DC calls per transaction 
average processor time per transaction 
largest control block storage required 


X 
X 

X 
X 
X 
X 


X 
X 

X 
X 
X 
X 



Figure 1.2 Su^ested Factors to Include in Documentation of Workload 

The importance of fully documenting the workload cannot be overemphasized. 
Some of the most significant performance gains to be achieved in MVS are 
accomplished by means of workload management and the ability to manage the 
workload is directly proportional to detailed knowledge of the workload. 
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Step 4: Setting the Objectives 



Once the workload is categorized, set objectives for each category of work. 
Experience with some MVS systems shows that installations don't always know 
the specific response and turnaround requirements of different users. As a 
starting point, use the response or turnaround time achieved on the pre-MVS 
system. If such data was not formally reported, it is still probably available from 
old trace tapes, the output of old reduction programs, the operator log, or similar 
sources. Examine the objectives achieved on the pre-MVS system in light of user 
requirements and the priority of the work. If necessary, revise the objectives met 
on the pre-MVS system. 

Many installations state objectives for a percentage of the transactions in a 
class — for example, 90% of TSO transactions should receive a three-second 
response time; 85% of the jobs in class A should receive tumaround time of one 
hour. If you state objectives in these terms, also set an objective for the "left- 
over" percentage of transactions — in the preceding example, for the 10% of TSO 
transactions and the 1 5% of jobs in class A. For very heavy transactions or jobs, 
you might want to specify the objectives in terms of service units or processor 
time received per second. Ensure that objectives are set for 100% of the work in 
your system. 

In setting user-oriented objectives, be sure you take into account any time the 
user sees that is not reflected in the measurement of the objective. For example, 
if TSO trivial transactions require a four-second response time, you might set the 
objective to three seconds to account for polling delays not reflected in RMF or 
MF/1 measurementsof response time. 



Step 5: Measuring Resource Requirements 

Once MVS is generated and stable, measure the resources actually being used by 
the different categories of work. To do this, you must choose the means by 
which you will measure resource consumption (for example, service units, 
seconds, number of events such as the number of EXCPs, and so on) and the tools 
by which the resources will be measured. Essentially, you want to identify the 
amounts of processor, real storage, and I/O resources required for each category 
of work. Figure 1.2 includes resource requirements that might be measured. 

By assigning each distinct category of work to a separate performance group, 
you can obtain data on the processor, real storage, and I/O service units consumed 
by each category from the RMF workload activity report. Note, however, that 
there are two exceptions: (1) I/O service for IMS will not be reported, since all 
IMS EXCPs are issued in key 7 ; and (2) service will not be reported for privileged 
address spaces — that is, those address spaces for which you turned on the 
privileged bit in the PPT to make the address space nonswappable unless it enters 
long wait. Instead of using the privileged bit, place the address spaces in a domain 
where the minimum MPL is greater than or equal to the number of address spaces 
in the domain. The address spaces will not be swapped and RMF will report the 
service units they use. 
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If you are using the same batch job classes in MVS as were used in MVT or 
SVS, you might want to assign each class to a unique performance group with 
identical IPS performance objectives; this will help you observe how the job 
classes are serviced in MVS and might help you identify changes to the job class 
structure. 

Track the resource measurements for an extended period of time so that they 
encompass all variations in the workload. Job- and transaction-related data should 
be tracked both as an average and as a distribution, so that you identify 
exceptional conditions. Such exceptions will help you judge the effectiveness of 
your workload categories and the possible need for installation controls on the 
exceptional work. For example: 

• Batch jobs whose resource consumption places them in the top ten percent of 
their class in terms of resource usage might require reclassification. 

• If the resource data varies widely for a particular job class — that is, there is 
no distinct pattern — that job class might require redefinition or a tolerant 
performance objective. 

The resource data collected will further define the categories of work; from 
this data, you can set resource limits for each category - for example, one 
processor minute for each job in job class X. Once the resource limits for each 
class are understood, consider installation controls and procedures to track and 
enforce these limits. Some resource limits, such as elapsed time and resource 
allocation, can be enforced using SMF exits. Others, such as number of EXCPs or 
amount of real storage, cannot be enforced while the program is running. You 
might want to produce exception reports that list jobs/transactions that exceed 
any one of the resource limits. Or adjust billing rates for jobs that exceed the 
resource Hmits for their class, to provide an incentive to the user to correctly 
classify his work. 

Understanding the resources required for each category of work, at each level, 
will help you to judge the reasonableness of your objectives and to set capacity 
limits for each category: the throughput measure which, if exceeded, will impact 
other work in the system. The following formulas might be useful in helping you 
determine capacity limits for batch and TSO work: 

TSO processor % = average processor seconds per TSO transaction x 
transaction rate x 100 

, ^ . ^ 100 X average processor seconds per job x number of jobs 

batch processor % = . ■ \r ' , . ' i — — 

*^ seconds m the measured mterval 

Figure 1.3 illustrates a complete performance objective for a batch job class, 
including the definition of the category, resource requirements, and capacity 
limits. 
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Batch class = 6 

Prime shift: 8.00 - 6.00 

Class description: short duration batch 

Objectives for class: 
class turnaround 



90% of the jobs will clear the printer in less than 30 minutes 
(from first card read) 



throughput limit — 400 jobs 
class resource consumption limits 



for each job in the class 



processor minutes 




1.00 


EXCPs 




1000 


elapsed time (minu 


tes) 


10 


storage requested 




512K 


disk mounts 




none 


tape mounts 




none 


cards in 




1000 


cards out 




500 


lines printed 




4500 


scratch space 




25cyl 


contiguous scratch 


space 


5cyl 



Figure 1.3 Sample Performance Objective for Batch Job Class 
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Chapter 1.2 Selecting Measurement Tools 



The goal of any performance evaluation is to make the most effective use of the 
basic system resources: processor, storage, channels, I/O devices, and control 
units. In order to do this, it is necessary to first determine the specific 
performance problems that might exist in the system by examining the current 
use of the resources. Once potential solutions are identified and implemented, it 
is necessary to re-evaluate resource utilization to judge the effectiveness of the 
solutions. Measurement tools are the means to determine how resources are being 
used; any performance analysis effort is, in fact, limited by the software and 
hardware measurement tools that are available to the installation. 

The purpose of this chapter is to present an overview of tools available from 
IBM and additional techniques that can be used to gather data and measure 
resource usage in the system. This chapter is divided into the following topics: 

• "MVS Measurement Tools," which describes the types of data provided by 
various tools. 

• "Other Sources of MVS Data," including control blocks, the trace table, and 
SYSl.LOGREC. 

• "Writing Data Reduction Programs," which focuses on extracting performance- 
related information from SMF and GTF. (Writing data reductions programs to 
extract information from control blocks and the trace table is described in 
"Other Sources of MVS Data.") 

• "Monitoring Performance," which describes the kind of data you should 
monitor on a continuous basis, so that you can recognize potential 
performance problems before they become crises. 



MVS Measurement Tools 



While SMF, GTF, and MF/1 (or RMF) are important IBM measurement tools that 
exist as part of the MVS system (or as a program product), the installation will 
usually need additional tools to collect and process all of the data required for a 
performance evaluation. A number of programs have been written to provide 
details both system-wide and by address space on system operations such as 
paging, real storage utilization, and processor utihzation. These are the currently 
available Installed User Programs (lUPs) and Field Developed Programs (FDPs). 
Most of these programs are data reduction tools for SMF or GTF. Others are 
software monitors, which use their own monitoring facilities to extract data and 
their own reduction capabilities to format the data. Many of these tools can and 
should be used together (either concurrently or sequentially) to get the most 
reaHstic and comprehensive picture of system resources as possible. 
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The following figures provide a comparison of various measurement tools 
available from IBM and of hardware monitors. {Note: IBM does not market 
hardware monitors. Any statement about their capabilities is based upon IBM's 
general knowledge of hardware monitors. While the statements are believed to be 
true, customers using hardware monitors should rely on their vendors for 
information.) 

• Figure 1.4 gives a general description of each tool. 

• Figure 1.5 lists input to and output from the tools. 

• Figure 1.6 notes operating characteristics and limitations of the tools. 

• Figure 1.7 summarizes the types of data provided by the tools. 

{Note: IMS and JESS tools are included only in figure 1.4.) In addition, figure 
1.8 compares the types of data provided by three software monitor programs: 
SUDP, SIRTSO/SIRBATCH, and JIA. Figure 1.9 lists the major system resources, 
the types of data that should be examined for each resource, and the key tools 
and/or control blocks that can be used to collect the data. (More information on 
control blocks is included in the next topic, "Other Sources of MVS Data.") 
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Tool 


Type 


General Description 


Documentation 


System Management 
Facilities (SIWF) 


MVS component 


collects a large amount of data, a subset of which 
is performance-related. Emphasis is on job-related 
data. 


0S/VS2SPL: System 
Management Facilities, 
GC28-0706 


Generalized Trace 
Facility (GTF) 


MVS component 


collects a large amount of data; traces system 
events. 


0S/VS2SPL: Service 
Aids, GC28-0674 


System Activity 
Measurement Facility 
(MF/1) 


MVS component 


collects system-wide information of a general 
nature; most is performance-related. Evaluates 
hardware performance. 


0S/VS2SPL: Initializa- 
tion & Tuning Guide, 
GC28-0681 


Resource 

Measurement Facility 
(RMF) 


program product 


same as MF/1 with more system data, two 
additional reports, and an offline post processor. 


0S/VS2 MVS Resource 
Measurement Facility, 
GIM,GC28-0736 


MVS Storage Utilization 
Display (SUDP) 


software monitor 
(FDP#5798-CGT) 


provides information on the use of real storage 
by particular jobs. 


SB21 -1753 and 
LB21-1754 


System Information 
Routines (SIR) 


software monitor 
(IUP#5796-PGB) 


SIRTSO — runs as a TSO command; provides 
enough job-related data to give a full picture of 
SRM and paging activity. 

SIRBATCH — runs as a batch job or started task; 
collects system and address space data at fixed 
intervals. Provides more details than SIRTSO. 


SH20-1813 


SVS/MVS System and Job 
Impact Analyzer (JIA) 


software monitor 
(IUP#5796-AJF) 


provides averages of various system activities on 
an address space basis. 


SH20-1720and 
LY20-2217 


MF/1 Post Analyzer 
(POSTANAL) 


SMF analysis program 
(FDP#5798-CHX) 


provides general system-related data via two 
report programs: MF1DFERD (Assembler 
Language) and MF1ANLZR (PL/1). More useful 
for capacity planning than for tuning. 


SB21-1814and 
LB21-1815 


OS/VS Capacity 
Management Aid (CMA) 


SMF analysis program 
(FDP#5798-CJB) 


provides an overview of the usage of various 
system functions via two Assembler and five 
FORTRAN programs. Useful for capacity 
planning. 


SB21 -1835 and 
LB21-1836 


GTF Supervisor Services 
Analyzer (GTFSVC) 


GTF analysis 

program 

(IUP#5796-PGE) 


provides details about SVC usage on a system and 
job basis. Should be considered one of the major 
tuning tools in situations where supervisor time 
is critical. 


SH20-1816 


MVS Seek Analysis 
Program (SEEKANAL) 


GTF analysis 

program 

(IUP#5796-PJC) 


provides details about SVC and module usage and 
about seek operations on DASD on a system or 
job basis. Should be considered one of the major 
tuning tools. 


SH20-1814 


GTF I/O Concurrency 
(GTFIOCUR) 


GTF analysis 

program 

(IUP#5796-PGD) 


provides details about potential device contention 
resulting from concurrent operations on channels 
and control units; used to minimize DASD I/O 
interference. 


SH20-1815and 
LY20-2240 


GTF Direct Access 
Contention Analyzer 
(DACA) 


GTF analysis 

program 

(FDP #5798-CEZ) 


provides details about the interference due to 
accessing shared DASD and interference between 
devices under the same control unit. 


SB21-1654 


GTF VTAM Buffer Trace 
Analysis (GTFVTAM) 


GTF analysis 

program 

(IUP#5796-PGF) 


provides information about the utilization of 
VTAM buffer pools to aid in designing the right 
size pool. 


SH20-1817 


GTF Data Analysis 
Program (GTFANAL) 


GTI^ analysis 

program 

(FDP#5798-CHT) 


handles all types of GTF records; reports seek 
statistics and statistics about arm movement. 


SB21-1808 


Generalized Data Area 
Monitor and Display 
Program (Data Area 
Monitor) 


(FDP#5798-CKK) 


monitors and displays user or system data areas 
in OS/VS system; data areas to be monitored 
are indicated by means of control cards. 


SB21 -1910 and 
LB21-1911 



Figure 1.4. General Information on MVS Measurement Tools (Part 1 of 3) 
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Tool 


Type 


General Description 


Documentation 


Hardware Monitors 


hardware monitor 


captures and records electrical signals created by 
the system during normal operation. 




JESS Monitoring 
Facility (JMF) 


software monitor 
(IUP#5796-PHR) 


monitors activity within the JES3 address space on 
an MVS global processor and records data on the 
following: JESS prpcessor/address space activity; 
JESS FCT activity; JESS spool data management; 
JESS device scheduling activity; and JESS job 
throughput. 


SH 20-1 881 


IMS/VS Log Transaction 
Analysis Utility Program 
(DFSILTAO) 


IMS component 


based on information from the IMS log tape 
about individual occurrences of IMS/VS trans- 
actions, calculates total response time, time on 
the input queue, processing time, and time on 
the output queue. Produces a new IMS/VS log 
tape, if specified; a report on disk that can be 
sorted to produce a sequenced report; a detailed 
report of transactions in input sequence^ if 
specified; and a summary report. 


IMS/VS Utilities 
Reference Manual, 
SH 20-9029 


IMS/VS Statistical 
Analysis Utility Programs 


IMS component 


Using the IMS log as input, produces a list of 
messages in line and terminal sequence, a list 
of messages in transaction code sequence, or 
statistical reports. Reports include the following: 
messages queued but not sent; program-to- 
program messages; line and terminal report, 
which shows line and terminal loading by time 
of day; transaction report, which shows loading 
by transaction code and by time of day; trans- 
action response report; application accounting 
report, which includes TCB processor time and 
counts of all DL/I requests; and IMS/VS 
accounting report, which shows start and stop 
time for the control region. 


IMS/VS Utilities 
Reference Manual, 
SH 20-9029 


IMS/VS Program Isolation 
Trace Report Utility 
Program (DFSPIRPO) 


IMS component 


Lists all transactions that had to wait while trying 
to enqueue on a data base record; prints the 
waiting transaction, the holding transaction, and, 
if the /TRACE ALL option is used, the elapsed 
time of the wait. 


IMS/VS Utilities 
Reference Manual, 
SH 20-9029 


IMS DC Monitor 


IMS component 


provides key values such as elapsed time, IWAIT 
time, elapsed processor time, scheduled time to 
first DL/I call, elapsed execution time, and queue 
statistics. The IMS/VS Monitor Report Print 
Program (DFSUTR20) summarizes and categorizes 
the information at various levels of detail, pro- 
ducing the following reports: system configuration; 
statistics from buffer pools; region summary; 
region IWAIT; programs by region; program 
summary; program I/O; communication summary; 
communication IWAIT; transaction queueing; 
DL/I call summary; and distribution appendix. 


IMS/VS Utilities 
Reference Manual, 
SH20-9029 



Figure 1.4. General Information on MVS Measurement Tools (Part 2 of 3) 
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Tool 


Type 


General Description 


Documentation 


IMS Monitor Summary 
and System Analysis 
Program (IMSASAP) 


(FDP#5798-CDT) 


a set of programs designed to process DFSTRAPC 
output from IMS/VS Monitor (IMS/VS 1.0.1). 
It uses a subset of the data collected by 
DFSTRAPC to produce several selectable 
additional reports designed to fill the needs of 
management, system analysts, and programmers. 


SB21-1582 


IMS Log Tape Analysis 


(FDP #5798-CAQ) 


provides information on response times and 
message traffic. 


SB21-1402 


IMS/VS Virtual Storage 
Analysis 


GTF reduction 

program 

(FDP#5798-CNC) 


provides an IMS/VS memory map, showing the 
name, real storage locations, and length of various 
pools and modules in the IMS/VS system, and 
three page fault activity reports designed to trace, 
summarize, and categorize page faults associated 
with the IMS/VS system. 


SB21 -2003 


IMSMAP/VS Data Base 
Mapping Programs 


(IUP#5796-PCY) 


provides two data base mapping programs: 
DBDMAP, which builds and prints maps of IMS 
physical and logical data bases and a descriptive 
report of each data base; and PSBMAP, which 
builds and prints maps of IMS physical and logical 
data bases associated with program specification 
blocks. Note: IMSMAP (lUP #57960-PBC) is a 
prerequisite to the installation of IMSMAP/VS. 


SH 20-1 539 



Figure 1.4. General Information on MVS Measurement Tools (Part 3 of 3) 
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Tool 


Input Data 


Needs Data 
Reduction 
Program ? 


Format of Output 


Overhead 


SMF 


control blocks and 
data areas 


YESl 


from 1 to 76 record types 


low to moderate depending on the level of 
information and the type of system activity. 
I/O can be high if large numbers of records 
are produced. 


GTF 


control blocks and 
data areas 


YESl 


trace records and control 
records 


moderate to high depending on the number 
and type of events being traced. 


MF/1 


control blocks and 
data areas 


NO 


formatted printed records 
and/or SMF records 70-74 


biggest impact is during report generation; 
the shorter the interval between reports, the 
bigger the burden on the system. Intervals 
of 15 minutes or more are recommended. 


RMF 


control blocks and 
data areas 


NO 


formatted printed records 
and/or SMF records 70-76 


low; the post processor eliminates the high 
overhead. 


SUDP 


control blocks 


NO 


real-time displays; bar 
graphs for a 3277 or on 
hard copy 


load on system is light (2-3%) if interval is 
5 seconds or more. 


SIRTSO 
SIRBATCH 


control blocks 


NO 


real-time display on 3277 
TSO terminal and optional 
hardcopy 


low; these programs do not use locks to 
protect themselves from control block 
changes during execution. 


printed reports; report 
routines edit data 


JIA 


control blocks: 
CVT, ASVT, 
ASCB.OUXB, 
OUCB, CSCB 


NO 


bar charts; report routines 
edit data 


low for a 5 second interval. 


POSTANAL 


MF/1: SMF 
records 70-74 


NO 


compact listing (no 
carriage controls), 
summary reports and 
histograms 




CMA 


SMF records 
0,5,12,35 
MF/1: SMF 
records 70-74 


NO 


listings and graphs 




GTFSVC 


GTF: SVC 
records 


NO 


summary and detail 
reports 


real storage usage is relatively high, and a 
lot of sorting may take place. 


SEEKANAL 


GTF: SVC 
and/or SIO 
records 


NO 


reports and graphs 


execution fast; no external sorting. 


GTFIOCUR 


GTF: SIO and 
I/O records 
with timestamps 


NO 


reports and graphs 


substantial 


DACA 


GTF: SIO 
and I/O 
records 
with time- 
stamps 


NO 


1-page report 
per device 


substantial 


GTFVTAM 


VTAM SMS 
trace records 
(collected via 
GTF with the 
USR option) 


NO 


tabular summary reports 


execution fast 


^Note, however, that SMF and GTF data can also be used without summarization. 
SMF is an audit trail of system content and workload. GTF can be used for investigating 
transacrtion-related performance problems without data reduction. 



Figure 1.5. Input to, Output from, and Overhead of MVS Measurement Tools (Part 1 of 2) 
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Tool 


input Data 


Needs Data 
Reduction 
Program ? 


Format of Output 


Overhead 


GTFANAL 


GTF: DSP, 
EXT, PCI, 
SRM,SIO,l/0, 
SVC, and RR 
records 


NO 


broad range of reports 
available fronn single run 


number of output pages is high 


Data Area Monitor 


data areas 
selected by 
installation 


NO 


detail report, each line of 
which consists of a user- 
specified description and 
as many observations as 
will fit on the print line; 
and summary report, 
which displays the 
summary data for each 
of the user-defined 
variables. 


low 


Hardware 
Monitors 




NO 


real-time measurements; 
data is summarized and 
written to tape; programs 
format it into graphs. 


low; they might not interrupt processing 
if run independently of system processor 
time. 



Figure ITS. Input to, Output from, and Overhead of MVS Measurement Tools (Part 2 of 2) 
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Tool 


Operating Characteristics 


Limitations 


Notes 


SMF 


• parameters specify record types and 
exit routines. 

• SIVIF buffer size siiould be in 4K- 
8K range in order to prevent 
excessive I/O and ENQ delays. 

• use 1 EFU83 exit routine to avoid 
writing unnecessary records, but 
retain the capability to periodi- 
cally sannple all work in terms of 
storage, EXCPs, blocksizes, etc. 

• turn off unused exits. 


No data collected for system tasks or 
problem programs started from console 
or running in system keys. Data can be 
collected for a subsystem only if it 
is started as a job in the form of a card 
deck or invoked through an internal 
reader procedure. 


• should be used with MF/1 or 
RMF. 

• should be used to locate 
dominant jobs, programs, 
and files. 


GTF 


• options specify the system events 
to be traced. 

• run at a high dispatching priority 
to reduce lost events since it is 
generally used for short spurts of 
system activity. 


Does not provide environmental data 
such as storage size, alternate paths, 
and volume serial numbers. Issue the 
DISPLAY UNITS command to obtain 
the device address and the volume 
currently mounted on the device. 


In response or turnaround situ- 
ations, time stamping should be 
used. 


MF/1 and RMF 


• options specify the reports and 
intervals to be generated, 

• run in RECORD mode to collect 
data in machine readable form and 
have it written to the active SMF 
data set. 

• to trace individual jobs, assign each 
to its own performance group. 

• put each IMS MPR in a different 
performance group to get the 
processor utilization, absorption 
rate, and task time for each one. 


• only one can be active at a time; 
must be stopped and restarted 
to change options, 

• does not recognize the type of 
work being run on the system; 
however, you can obtain data on 
distinct categories of work by 
assigning different types of work to 
different performance groups. 

• does not report services of 
privileged address spaces; instead 
of using the privileged bit in the 
PPT, assign "privileged" address 
spaces to a domain where the 
minimum MPL is greater than or 
equal to the number of address 
spaces in the domain to ensure 
they are not swapped. 


• can be used with SMF to 
obtain a complete picture 
of system workload (see 
SPL: initialization and 
Tuning Guide). Use these 

two tools to compare paging 
rates, I/O activity, and services 
of a problem program and the 
system. 

• RMF has a post processor 

to read in RMF SMF records 
and print reports. 


SUDP 


must run non-swappable and 
authorized. 

• monitor interval and display types 
can be changed by a light pen. 


Does not run under TSO. 




SIRTSO 


• subcommands specify the address 
spaces to gather information for: 
batch , TSO-users, swapped-in, or all. 

• re-entrant program; can be placed 
inSYSI.LPALIB. 

• uses TPUT full-screen facility; 
specify FULLSCR=YES in the 
TCAM MCP. 




Used primarily to supplement 
data provided by MF/1 or RMF 
with detailed information about 
specific address spaces. 


SIRBATCH 


Must reside in an authorized library. 



Figure 1.6. Operational Considerations for MVS Measurement Tools (Part 1 of 2) 
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Tool 


Operating Characteristics 


Limitations 


Notes 


JIA 


Monitor program must reside in an 
authorized library; report programs 
can reside in private libraries. 


Non-swappable address spaces are 
incorrectly interpreted as V=R regions. 




POSTANAL 


Output volume can be controlled by 
selecting only certain reports. 


Reports and histograms are provided 
only for the total period covered by 
theSMF input data. 




CMA 


MF/1 records must be continous and 
taken at 10 minute intervals, or data 
can be misinterpreted. 


Output graphs require additional 
manual editing. 




GTFSVC 


Output volume can be controlled by 
selecting certain jobs, modules, or 
SVC; to cut overhead, only key jobs 
should be selected. 




Although time-stamped GTF 
records are not necessary, it 
may be useful to have a stan- 
dard format for GTF records; 
most other tools use them with 
time stamps. 


SEEKANAL 


SYSIIM parameters specify the types 
of reports and the intervals to be 
covered. 




The module activity report 
provides valuable input for 
creating a pack list. 


GTFIOCUR 


SYS IN parameters specify the types 
of reports and the intervals to be 
covered. 


Lost events will affect the validity 
of the data. 




DACA 




Shows the control unit busy state as 
seen by only one system. 




GTFVTAM 


• close GTF input tape properly to 
avoid wrong start/stop times. 

• GTF tracing with time stamps is 
recommended. 


Not valid with VTAM II 




GTFANAL 


option card specifies the reports 
desired. 






Data Area 
Monitor 


executes as a batch job with control 
cards. 


Can monitor/display only data areas 
accessible by problem program. 




Hardware 
Monitors 




Does not recognize the type of work 
being run on the system; must use 
SCP and job-related data to determine 
which jobs are causing the utilizations. 





Figure 1.6, Operational Considerations for MVS Measurement Tools (Part 2 of 2) 
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Tool 


Types of Data Provided 


SMF 


job-related: 

Type 4/34 records — swap counts, page-in/page-out counts, EXCP counts (including VIO accesses), processor 
time and storage used for a step. 

Type 5/35 records - Processor time under TCBs and SRBs, and job service unit count. 

Type 14/1 5 records — EXCP counts and other information for data sets. 

Type 26 record - actual SYSIN/SYSOUT line and card count. 

system-related (collected during MF/1 or RMF interval): 

Type 70-76 records — processor, paging, workload, channel, device, page-swap data set, and trace activity. 


GTF 


system-related: 

SIO trace records — shows seek activity on a device and S 10 operations. 

SVC trace records — identifies type, quantity, and users of supervisor services, such as ENQ/DEQ activity 
(including the resource name). 

SRM trace records - SRM activity and SYSEVENTS. 

I/O trace records — shows I/O interrupts by device. 

DSP trace records — shows all units of work dispatched by the system. 

PI trace records — program interrupts. 

EXT trace records — external interrupts (clock and signal processor interrupts). 


MF/1 


system-related (collected during each interval): 

CPU Activity Report — processor wait time. Processor utilization = 100% — wait percentage. 

Channel Activity Report — number of successful SIOs issued to each channel, percent of time channel 

busy, percent busy while processor waited. 

I/O Device Activity Report — number of successful SIOs issued to each device, percent of time device busy, 

average number of requests enqueued on each device. 

Paging Activity Report — details about paging demands, a snapshot of real and auxiliary page storage, 

and swapping statistics. 

Workload Activity Report — information on workload activity for each performance group period and for 

the system as a whole. 


RMF 


Same as MF/1 , plus the following: 

additional data in the Paging Activity Report — SRM swap out counts, sampled minimum, maximum, and 

average real and auxiliary page storage. 

additional data in the Workload Activity Report — IOC, CPU, and MSO service swaps by performance group 

and domain numbers. 

Page/Swap Data Set Activity Report — statistics on individual data set usage. 

ASM/RSM/SRM Trace Activity Report - sampled values of fields from PVT and ASMVT control 

blocks, SRM Data Area, and SRM Domain Tables. 


SUDP* 


job-related: 

Real Storage utilization — percent used/fixed by each job, TSO user, and system task, and a swap indication, 
system-related: 

Paging Rates — demand paging, VIO paging, and swapping. 

Internal Queues of Frames — available pages, common, SQA, LSQA, and local frames and their status. 


*See Figure 1.8 for a more detailed comparison of the data provided by SUDP, SIR, and JIA. 



Figure 1.7. Types of Data Provided by MVS Measurement Tools (Part 1 of 3) 
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Tool 


Types of Data Provided 


SIRTSO* 


job-related : 

paging and swapping, frame usage, OUCBs. 
system-related: 

processor utilization, available queue depth, IPS id and UIC value. 


SIRBATCH* 


job -related: 

paging and swapping, page frames, OUCBs (for specific job, for swapped-in address spaces, first 40 address 
spaces, or all address spaces). 

system-related: 

page frames and OUCBs, paging and swapping, actual EIMQ contentions (for all or system-wide only). 


JIA* 


job-related: 

Processor utilization — percentage over a defined summary period. 

Real storage — allocation broken down by the various frame queues, 
system-related: 

Average processor utilization, EXCP rate, average real storage per address space over a defined period. 

Page I/O — option to print only intervals with paging above a given threshold. 


POSTANAL 


system-related: 

Summary reports with averages and totals — processor, paging, workload, channel, and device activity. 

Histograms — 

• percentage of real storage frames • utilization per processor, 
allocated. • channel utilization. 

• swaps per minute. • percentage of transactions ending 1st period. 

• percentage of pagespace slots allocated. • ratio of ended transactions per number of swaps. 

• SIOs per second. • non-swap paging rate. 

• average number of pages per swap sequence. 

• service by performance group. 


CMA 


system-related: 

Processor load — for batch, TSO, or total. 
Storage usage — swapping and paging. 
Application development — job steps and TSO sessions. 
Percent of week processor is active. 
Daily load profile. 


GTFSVC 


job-related: 

Job summary report — SVC usage per job in percent of total SVC usage. 

Job detail report — SVC usage for a specified job by TCB address, by module name, and by PSW address, 
system-related: 

System summary report — SVC usage by number. 

SVC detail report — usage of SVCs by jobname. 


SEEKANAL 


job or system-related: 

SVC usage report — usage count per SVC. 

Module usage report — usage count per module as given by SVC type records. 

Start I/O report — number of SIOs per device, broken down by condition codes. 

Seek histogram report — number of seeks per DASD broken down by seek address. 

Module affinity report — how often control was transferred between certain modules. 


*See Figure 1.8 for a more detailed comparison of tlie data provided by SUDP,SIR,and JIA. 



Figure 1.7. Types of Data Provided by MVS Measurement Tools (Part 2 of 3) 
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Tool 


Types of Data Provided j 


GTFIOCUR 


system-related : 

Device I/O concurrency — matrix of absolute and relative concurrent activity. 

Control unit I/O concurrency — concurrent activity within and between control units. 

Channel I/O concurrency — absolute and relative concurrent activity. 

Device I/O overlap — how much time was overlapped with the operation of the other devices, in 

absolute and relative terms. 

Condition code analysis — total count of SIOs per device (broken down by condition codes) and I/O 

interrupts per device. 

Recommendation for volume — shows for each pair of devices, how much the potential interference would be 
swaps increased or decreased if they were interchanged. 


DACA 


system-related: 

1 -page report per device — control unit busy in seconds and percentage of total time, device busy, control 
unit and device busy, no contention. 
Note: Device busy times measure interference due to sharing devices between processors. Control unit busy times 
measure interference between devices under the same control unit. 


GTFVTAM 


system-related: 

Run summary report — start/stop times for VTAM and GTF, number of records processed. 

Buffer pool summary — number of elements available on average, high, and low use values, number of 
report (for each VTAM times that the specified threshold was exceeded, 
buffer pool) 

Sequential SMS trace — pertinent trace data including jobnames and time stamps in chronological order, 
listing 

Mode summary report — number of elements available per buffer pool ordered by their occurrences. 


GTFANAL 


job-related: 

Jobname detail — all events in detail within jobname. 

Jobname summary — all events by type within jobname. 

SVC usage — by jobname within SVC number. 

I/O usage — I/O interrupts by jobname within I/O device. 

SIO usage — by jobname within I/O device. 

Jobname miscellaneous (6 reports) — external interrupts, dispatcher, page faults, set system mask, PCI, and 

module usage within jobname. 

SRM detail — events by address space, 
system-related: 

Module analysis — counts LINK, LOAD, XCTL, and ATTACH SVCs by jobname within module name. 

SIO analysis — condition codes and channel status word by device. 

I/O analysis — channel status word bytes for I/O interrupts by device. 

SRM event summary. 

Recovery routine list — accesses to recovery routines. 

Disk seek analysis — analysis of seek addresses and arm movement per device. 


Data Area 
Monitor 


system-related: 

any system data areas accessible by problem program. 


Hardware 
Monitor 


system-related: 

Processor utilization — processor active time, time in problem program state vs. supervisor state, number 
of instructions executed. 

I/O utilization — channel and control unit busy time. 

Processor and I/O overlap. 



Figure 1.7. Types of Data Provided by MVS Measurement Tools (Part 3 of 3) 
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SIRTSO 


SIRBATCH 


SUDP 


JIA 


general information 


no 


yes 


yes 


yes 


ASID 


jobname 


yes 


yes 


yes 


yes 


time of snapshot 


yes 


yes 


yes 


yes 


storage usage 










by system queue 


no 


yes 


yes 


yes 


by address space 


yes 


yes 


yes 


yes 


fixed per address space 


no 


yes 


yes 


no 


processor usage 










total system 


yes 


yes 


no 


no 


per address space 


yes 


yes 


no 


yes 


paging 










total system 


no 


yes 


yes 


yes 


per address space 


yes 


yes 


no 


no 


swapping (per address space) 










swap status 


yes 


yes 


yes 


no 


number of swaps 


yes 


yes 


no 


no 


swap size 


yes 


yes 


no 


no 


other SRM data 










available queue 


yes 


yes 


no 


no 


service received 


yes 


yes 


no 


no 


UIC 


yes 


yes 


no 


no 


ENQ contention 












no 


yes 


no 


no 



Figure 1.8. Comparison of Data Provided by SIRBATCH/SIRTSO, SUDP, and JIA 
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Data Collected for System Resource 


Tools to Use 


Control Blocks to Use 


/. Processor Time 






for total system 


MF/1 or RMF 

SMF record 70 

SIR 

JIA 

POSTANAL 

CMA 

Hardware Monitor 


CCT (processor 
management control 
table) 
RMCT 


for supervisor activities 


GTF SVC record 
SEEKANAL 




for individual jobs 


SMF records 4/34 (for step) 
SMF records 5/35 (for job) 
SIRBATCH 
JIA 


ASCB 


for installation-defined groups 
of work (CPU service) 


RMF 




//. Storage Usage 






for total system (real) 


RMF 

SIRBATCH 
JIA 
CMA 


PVT 
PFT 


for individual jobs (real) 


SUDP 

SIRBATCH 

JIA 


ASCB 


for installation-defined groups 
of work (MSO service) 


RMF 




for IMS 


IMS/VS Virtual Storage 
Analysis 




individual job/job step working 
set size 


SIRBATCH 


SPCT 
OUCB 


for job steps (virtual) 


SMF re cords 4/34 




nucleus (real) 


RMF 
SUDP 


PVT 


SQA (real) 


SUDP 
SIRBATCH 


PFT 


LPA and CSA (real) 


SUDP 
SIRBATCH 


PFT 


common area (virtual) 


RMF 
SUDP 


PVT 


common area paging 


MF/1 or RMF 
SMF record 71 
SIRBATCH 


PVT 


total system paging 


MF/1 or RMF 

SMF record 71 

SUDP 

SIR 

JIA 

POSTANAL 


PVT 
RMCT 



Figure 1.9. Sources of Data on System Resources (Part 1 of 2) 
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Data Collected for System Resource 


Tools to Use 


Control Blocks to Use 


//. Storage Usage — (continued) 






IMS paging 


IMS/VS Virtual Storage 
Analysis 




job step swap size 


RMF 


PVT 




SMF records 4/34: 


OUCB 




pages swapped 






total number swaps 




individual job swapping 


SIR 




job workload level 


SIRTSO 


OUCB 


///. I/O Activity 






channels 


MF/1 orRMF 
SMF record 73 
GTF SI 0-1/0 records 
POSTANAL 
GTFIOCUR 
Hardware Monitor 


RMCT 


devices 


MF/1 orRMF 
SMF record 74 
GTF SI 0-1/0 records 
POSTANAL 
SEEKANAL 
GTFIOCUR 
DACA 
GTFANAL 




control units 


GTF SI 0-1/0 records 

GTFIOCUR 

DACA 

Hardware Monitor 




volume — tape 


MF/1 orRMF 
SMF records 19/21 




volunne - DASD 


SMF records 14/15 
MF/1 orRMF 
GTF SIO record 
SYS1.L0GREC 




EXCPsforjob 


SMF records 4/34, 14/15, 
40,64 




for installation-defined groups 


RMF 




of work (IOC service) 






EXCPs for system 


JIA 
GTFSVC 





Figure 1.9. Sources of Data on System Resources (Part 2 of 2) 
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other Sources of MVS Data 



Be aware that the measurement tools described in the previous section do not 
necessarily provide a complete evaluation of the current performance of a system. 
They can not totally determine how and under what conditions each resoiirce is 
being used, nor can they provide information about the existing system configura- 
tion while the data is being collected. It is therefore important to use a number of 
techniques to get information about the system. Additional sources of informa- 
tion on the system environment and activities in the system include the following : 

• SYSGEN stage I listing 

• JESGEN listing 

• PARMLIB listing 

• procedure library listings 

• VTOC listings of on-Hne volumes 

• output of the DISPLAY M command for configuration information 

• output of the display units command for on-line DASD and their attributes 
(d u, dasd, online) 

• the SYSl .LOGREC data set 

• system control blocks 

• the system trace table. 

This section describes performance-related data that can be derived from control 
blocks, the trace table, and the SYSl. LOGREC data set. 

Control Blocks 

System control blocks, available for direct examination while the system is 
operating, contain descriptive and status information about work executing in the 
system. Most of the information resides in globally addressable storage and can be 
monitored on a periodic basis by a user-written data reduction program. A data 
reduction program can generate information from control blocks such as: 

• the identification of users that are swapped into real storage. 

• the amount of real storage being occupied. 

t how processor time is being apportioned to users ; and so on. 

In general, this type of information is contained in control blocks that are 
allocated on an address space basis, although system-wide information of possible 
interest is also available. 

System control blocks are especially useful for determining an address space's 
use of storage over a particular interval. The ASCB, RSM header, and OUXB 
contain frame counts on an address-space basis that are dynamically updated; the 
SPCT also contains frame counts on an address-space basis, but the counts are 
updated only when an address space is swapped out. The PVT contains system- 
wide frame counts. By monitoring data on an address space's storage usage, you 
can observe and analyze activities such as: 

• The address space has no allocated frames — it is swapped out. 

• The address space has fixed an unusually large number of frames and is tying 
up storage. 
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The following paragraphs describe some key fields within control blocks that 
are of potential interest in a performance evaluation. See the 0S/VS2 MVS 
Debugging Handbook for additional information about these control blocks. 
(Note: The fields might be release-dependent; be aware that changes in control 
block fields can impact the results of your data reduction programs.) Figure 
1. 10 illustrates the chaining of several performance-related control blocks; figure 
1.9 lists control blocks that provide information on the major system resources. 



PVT 



PFT 



CVT 



4 PFT 



if ASVT 



I ASMVT 



4 RMCT 




ASVT 



ASCB, 



ASCB, 



ASCB 



ASCB 




ASCB, 



I RSM 



I OUCB 



I OUXB 




OUCB 



OUXB 



Figure 1. 10. Performance-related Control Blocks 
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Page Vector Table - PVT 

The PVT, pointed to by the CVT, contains the following system-wide data that 
can be of interest to system performance : 

PVTAFC Available frame count. 

PVTNPIN to These fields contain counters of system paging ; the sum of these 
PVTCAVR fields provide a count of the total system paging. 

PVTCFMCT Contains a count of the frames allocated to LPA and CSA. 

Page Frame Table — PFT 

The PFT, pointed to by the PVT, contains one entry for each real frame in the 
system, with the exception of the nucleus. Each entry identifies the following: 

• the "user" of the frame : the ASID if owned by an address space; X'FFFF' 
if a system frame; or logical group number, if last used for a VIO page. 

• the frame type. 

• the fix status. 

• the virtual address of the frame. 

• the unreferenced interval count (UIC). 

SRM Control Table - RMCT 

The RMCT is pointed to by the CVT. Following are several fields containing 
system-wide data of potential interest to performance: 

CCVUTILP Contains short-term processor utilization (PCT). 

CCVLGUTL Contains long-term processor utilization (PCT * 256). 

MCVPTR The non-swap paging rate. 

Auxiliary Storage Manager Vector Table - ASMVT 

The ASMVT is pointed to from the CVT and contains several system-wide items 
of potential interest : 

ASMSLOTS Contains a count of the number of slots in the paging subsystem. 

ASMVSC Contains a count of total slots allocated to VIO. 

ASMNVSC Contains a count of total slots allocated for non-VIO slots. 

ASMERRS Contains a count of slots unavailable due to I/O errors. 

ASMIORQR Contains a count of I/O requests received for processing by ASM. 

ASMIORQC Contains a count of I/O requests completed by ASM. 

ASMBURST Contains a burst value in microseconds (set to 50000) which is 
used to compute the maximum number of paging requests to 
place in a single channel program. 



1.32 0S/VS2 MVS Performance Notebook 



Address Space Control Block - ASCB 

The ASVT contains an entry for each potential address space. Each entry points 
to an ASCB, which contains job-related data. The following fields in the ASCB 
are of interest: 

ASCBSEQN The sequence number of this ASCB on the dispatching queue. 
Valid only if the address space is currently swapped-in. 

ASCBDP The current dispatching priority for this address space. Valid 

only if the address space is swapped-in. 

ASCBEJST This doubleword (in time-of-day clock format) represents the 

total task time received by this address space. 

ASCBSWCT Contains a count of the number of short waits issued by this 

address space. This value is used in the APG mean-time-to-wait 
calculation. 

ASCBVSC Contains a count of the total number of VIO slots allocated 

within the page data sets for this address space. 

ASCBNVSC Contains a count of the total number of non-VIO slots allocated 
within the page data sets to this address space. 

ASCBFMCT Contains a count of the number of real storage page frames 
currently occupied by this address space. 

ASCBJBNI Contains a pointer to the 8-character jobname for a batch job. 

Zero if not a batch job. 

ASCBJBNS Contains a pointer to the 8-character jobname for started tasks, 

mounts, and TSO users. 

ASCBSRBT This doubleword (in time-of-day clock format) contains the 
SRB time accumulated by this address space. 



Swap Communication Table - SPCT 

The SPCT indicates LSQA and fixed pages that are included in the working set for 
swapped-out address spaces, and contains other address-space-related swap 
information. 



SRM User Control Block - OUCB 

The ASCB for an address space contains a pointer to the OUCB. The OUCB is the 
key SRM control block for job -related data and contains valuable information 
about how the SRM sees the address space. The following fields are of primary 
concern in a performance evaluation : 

OUCBQFL This flag byte contains several indicators. OUCBOUT indicates an 
address space is swapped out but ready to run (on the OUT queue). 
OUCBOFF indicates the address space is swapped out but not 
ready to run (on the WAIT queue). 
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OUCBEFL This flag byte contains several indicators. OUCBLWT indicates 

that the address space is in a terminal wait. OUCBQSS/OUCBQSC 
indicate, respectively, that a quiesce has started and has been 
completed. (This field can indicate, for example, whether a TSO 
user is swapped out due to a long wait/terminal wait situation 
(normal) or due to SRM swapping — that is, if OUCBQSS and 
OUCBQSC are on with no indication of long wait or terminal 
wait.) 

OUCBSWC Contains a count of the number of swaps that have occurred during 
the life of the current transaction. 

OUCBPSO Contains the number of pages swapped out at the last swap-out of 
this address space. 

OUCBWSS Contains the number of pages swapped in at the last swap-in of this 
address space. In effect, this is SRM's view of the working set size 
of this address space; however, it is updated only at swap-out time. 

OUCBSRC This one-byte field is of interest in order to determine why the 
SRM swapped out an address space, as follows: 

binary value meaning 

'or output terminal wait 

'02' input terminal wait 

'03' long wait 

'04' auxiliary storage shortage 

'05' real pageable storage shortage 

'06' detected wait 

'07' requested by the address space 

'08' enqueue exchange 

'09' exchange on recommendation value 

'OA' unilateral swap-out 

OUCBWMR Contains the current normalized workload level for this address 
space. 

OUCBRMR Contains a recommendation value for the current resource use 
routines. 



System Resources Manager User Extension Block - OUXB 

The OUXB is also pointed to by the ASCB. It contains data about an address space 
that is not required by the SRM while that address space is swapped out. Analysis 
of this user-type paging data is also helpful when evjduating the performance of 
your system. 
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Trace Table Analysis 

Investigating some performance-related problems at the system level - such as 
resource contention, interlocks, and serialization — often depends on knowing the 
sequence of events in the system. Although it does not include all system activity, 
an analysis of the trace table via a user-written data reduction program can be very 
helpful in establishing a pattern for the system. By providing a sequential listing 
of an ordered set of events, the trace table characterizes the existing system on a 
more detailed level than what would be described by the SCP, hardware configura- 
tion, or workload. 

The major advantage of the trace table is that it provides information similar 
to GTF at a fraction of the cost in resources. Thus, you can gather real-time data 
at different periods of the day, thereby examining a variety of workloads in a 
production environment. In addition, overhead can be controlled by adjusting the 
frequency of trace samples taken. The more samples you can take without 
impacting the rest of the system, the more accurate your measurements will be. 
As a general rule, your sampling program should use less than 1% of processor 
time. A sample of 400 entries (a complete table) collected every 30 or 60 seconds 
over a 1 -2 hour period will present a fairly realistic picture of the system without 
impacting overall performance too heavily. 

The trace table, which is in the SQA, can be located via the address in location 
X'54'. Figure I.l 1 illustrates the format of the trace table. 



PSA 



Trace Table 



Loc X'54' 




header 



> 400 entries 



Figure I.l 1 . Trace Table Format 

There are eight types of entries that can appear in the trace table, each describing 
an event taking place in the system. The fifth hexadecimal digit in each entry 
indicates which type of event it is describing, as follows: 



digit 




type of entry 





SIO - 


- Start Input/Output 


1 


EXT - 


- External Interrupt 


2 


SVC - 


- SVC Interrupt 


3 


PGM - 


- Program Interrupt 


4 


ISD - 


- Initial SRB Dispatch 


5 


I/O - 


- Input/Output Interrupt 


6 


SSR - 


- Suspended SRB Redispatch 


7 


DSP - 


- Task Dispatch 



Figure 1.12 shows the format of each 32-byte entry. For examples and a more 
detailed description of trace entries, see 0S/VS2 SPL.MVS Diagnostic Techniques. 
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OFFSETS LENGTH NAME 



DESCRIPTION 



Conunon Ncune: Trace Table Entry 

Macro ID; None 

DSECT Name: None 

Created by; Not available 

Subpool and Key: Not available 

Size; Each entry is 32 bytes 

Function: The system trace table provides a record of 

events that have occurred. This trace table and its 

number of entries are an option that may be selected at 

system generation time. The following events cause entries 

in the system trace table: SIO Instruction? I/O 

Interruption; Program Interruption; External Interruption; 

each entry to the Dispatcher; SVC Interruption. 



OFFSETS LENGTH NAME 



DESCRIPTION 



(2) 



28 C1C) 



TRACE TABLE ENTRY TYPES 



SIO 

EXT 

SVC 
PGM 

ISD 

10 

SSR 
DSP 



START 



EXTERNAL 



TYPE CODE (CONTAINED IN BITS 
1,2, AND 3) 

(000) — SIO 
INPUT/OUTPUT 

(001) — EXT 
INTERRUPT 
(010) — SVC 
(Oil) — PGM 
INTERRUPT 
(100) — ISD 
DISPATCH 
CI 01) — I/O 
INTERRUPT 

(110) — SSR — SUSPENDED SRB 

REDISPATCH 

(111) -- DSP ~ TASK DISPATCH 



SVC INTERRUPT 
PROGRAM 



INITIAL SRB 



INPUT/OUTPUT 







ENTRY TYPE 


(SIO — START INPUT/OUTPUT) 




1 


(0) 
(1) 


1 
3 


CONDITION CODE 
DEVICE ADDRESS 


n 


t4) 


4 


CHANNEL ADDRESS WORD 


8 


C8) 


8 


CHANNEL STATUS WORD 


16 


(10) 


4 


lOSB ADDRESS FROM lOS 


20 
22 


C14) 
C16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
AS ID 


24 


C18) 


4 


TCB ADDRESS FROM SRB 



TIMER VALUE 



ENTRY TYPE 1 (EXT -- EXTERNAL INTERRUPT) 



EXTERNAL OLD PSW 
INTERRUPT CODE (BYTE 1 ) 
INTERRUPT CODE (BYTE 2) 
INTTIMER INTERNAL TIMER 
INTRPKEY INTERRUPT KEY 
..11 1111 EXTRSGNL EXTERNAL SIGNALS 

MALALERT MALFUNCTION ALERT (WITH 

INTERRUPT CODE (BYTE 1 ) BIT 6 
ON) 

1 EMERSGNL EMERGENCY SIGNAL (WITH 

INTERRUPT CODE (BYTE 1) 
BIT 6 ON) 
. . . 1 . EXTRCALL EXTERNAL CALL (WITH INTERRUPT 

CODE (BYTE 1) BIT 6 ON) 
. ..11 TODSYNCK TOD SYNC CHK 
. . 1 . . CLKCOMPR CLOCK COMPARATOR 
. .1.1 CPUTIMER CPU TIMER 




2 
3 


(0) 
(2) 
(3) 
1. . . 


8 

1 
1 




.1. . 
. .11 


iiii 



8 


(8) 


4 


REGISTER 15 CONTENTS 


12 


(C) 


4 


REGISTER CONTENTS 


16 


(10) 


4 


REGISTER 1 CONTENTS 


20 
22 


(14) 
(16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
AS ID 


24 


(18) 


4 


CURRENT OR TQE TCB ADDRESS 


28 


(1C) 


4 


TIMER VALUE 






ENTRY TYPE 2 


(SVC — SVC INTERRUPT) 





(0) 


8 


SVC OLD PSW 


8 


(8) 


4 


REGISTER 15 CONTENTS 


12 


(C) 


4 


REGISTER CONTENTS 


16 


(10) 


4 


REGISTER 1 CONTENTS 


20 
22 


(14) 
(16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
ASID 


24 


(18) 


4 


CURRENT TCB ADDRESS 



28 (1C) 



TIMER VALUE 



Figure 1.12. Format of Trace Table Entries (Part 1 of 2) 
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OFFSETS LENGTH NAME 



DESCRIPTION 



OFFSETS LENGTH NAME 



DESCRIPTION 







ENTRY TYPE 3 


(PGM — PROGRAM INTERRUPT) 





(0) 


8 


PROGRAM INTERRUPT OLD PSW 


8 


(8) 


4 


REGISTER 15 CONTENTS 


12 


(C) 


4 


REGISTER CONTENTS OR 
TRANSLATION EXCEPTION ADDRESS 


16 


(10) 


4 


REGISTER 1 CONTENTS 


20 
22 


(14) 
(16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
ASID 


24 


(18) 


4 


CURRENT TCB ADDRESS 


28 


(1C) 


4 


TIMER VALUE 






ENTRY TYPE 4 


(ISD -- INITIAL SRB DISPATCH) 





(0) 


8 


NEW PSW 


8 


(8) 


2 


ZERO 


10 


(A) 


2 


ASID 


12 


(C) 


4 


SRB ADDRESS OR CONTENTS OF 
REGISTER 


. 16 


(10) 


4 


PARAMETER LIST ADDRESS OR 
CONTENTS OF REGISTER 1 



20 (14) 
22 (16) 



28 (1C) 

Figure 1.12. 



CPU ID (4 LOW-ORDER BITS) 
ASID OF ADDRESS SPACE THAT 
CREATED THE SRB 



24 


(18) 


4 


PURGE TCB ADDRESS 


28 


(1C) 


4 


TIMER VALUE 






ENTRY TYPE 5 


(I/O — INPUT/OUTPUT INTERRUPT) 





(0) 


8 


I/O OLD PSW 


8 


(8) 


8 


CHANNEL STATUS WORD 


16 


(10) 


4 


RESERVED 


20 
22 


(14) 
(16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
ASID 


24 


(18) 


4 


CURRENT TCB ADDRESS 



TIMER VALUE 



ENTRY TYPE 6 (SSR ~ SUSPENDED SRB 
REDISPATCH) 





(0) 


8 NEW PSW 


8 
10 


(8) 
(A) 


2 ZERO 
2 ASID 


12 


(C) 


4 SRB ADDRESS OR CONTENTS OF 
REGISTER 


16 


(10) 


4 PARAMETER LIST ADDRESS OR 
CONTENTS OF REGISTER 1 



20 (14) 
22 (16) 



28 (1C) 



CPU ID (4 LOW-ORDER BITS) 
ASID OF ADDRESS SPACE WHERE 
SRB WILL BE DISPATCHED 



24 


(18) 


4 


PURGE TCB ADDRESS 


28 


(1C) 


4 


TIMER VALUE 






ENTRY TYPE 7 


(DSP ~ TASK DISPATCH) 





CO) 


8 


NEW PSW 


8 


(8) 


4 


REGISTER 15 CONTENTS (NEW) 


12 


CO 


4 


REGISTER CONTENTS (NEW) 


16 


CIO) 


4 


REGISTER 1 CONTENTS (NEW) 


20 
22 


C14) 
C16) 


2 
2 


CPU ID (4 LOW-ORDER BITS) 
ASID 


24 


C18) 


4 


NEW TCB ADDRESS 



TIMER VALUE 



Format of Trace Table Entries (Part 2 of 2) 
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Your sampling program should include some data reduction capability so that 
you can better interpret the data in the trace table. To help investigate 
performance-related problems and to aid in making meaningful correlations, each 
event traced is time -stamped, is related to a user (via his ASID), and can be sub- 
divided into specific events for a more detailed evaluation. Data reduction 
techniques can therefore be written to gather information and make calculations 
according to individual needs. The following list suggests several possible ways 
that the data from the trace table might be used in order to do a performance 
analysis: 

• Calculate the rate of occurrence of a particular event, such as an EXCP, or an 
ordered set of events, such as an EXCP followed by an SIO. (The example 
described later illustrates this type of calculation.) On an MP, it might be 
useful to analyze I/O activity for each processor, since symmetric devices 
might be impacting the overall configuration. Or, determine the probability 
that a successive event will occur immediately after a given event (for example, 
given an EXCP, what is the probability that the next event will be an SIO 
versus the I/O being queued?). 

• Determine how long a dispatched task executes before it relinquishes control 
voluntarily or is interrupted. This can be determined by taking, for a particular 
task, the difference between the time interrupted (via an I/O interrupt, page 
fault, wait, and so on) and the time the task was last dispatched. 

• Distinguish a task dispatch as being either a wait task, a supervisor task, or a 
problem program task by examining the CMWP bits in the PSW of entry type 7. 
Summary statistics can be computed to show the approximate problem 
program time and supervisor time under a task. If very few SRBs get 
interrupted, the SRB time can be approximated from the time of an SRB 
dispatch (entry type 4) to the time of the next event. If SRBs are interrupted, 
average SRB time can be approximated by summing the times from entry type 4s 
to the next event and the times from entry type 6s to the next event; and 
dividing this sum by the number of entry type 4s. 

• Map SVCs by address, job, or ASID. This enables a mapping of SVCs to 
modules and CSECTs and helps establish the distribution of supervisor services. 

• Collect elapsed time for individual SVCs by taking the difference between SVC 
events and SVC completion. SVC completion is indicated in dispatch entries 
where the SVC number is contained in the sixth, seventh, and eighth digits of 
the PSW; and where the PSW address, ASID, and TCB address are the same as 
the SVC entry. 

• Profile GETMAIN/FREEMAIN SVCs by size and subpool by examining 
registers and 1 . This will enable you to determine how storage is being 
allocated and unallocated. You can then evaluate the use of the GETCELL 
macro in place of GETMAIN if a particular user is using excessive processor 
time to do GETMAINs. 
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Identify which applications are initiating each event by examining the ASID 
in each entry and using control blocks to derive the jobname: 
ASID >-ASVT ► ASCB ► jobname 

Map system page faults by module. You can tell that a page fault was caused 
by a module or occurred when accessing the module by mapping the virtual 
address that caused the page fault against a NUCMAP or LPAMAP address. 
You can map just the page fault activity in the PLPA using the Linkpack 
directory. This information can be used to create a good packlist. 

Monitor for some specific event or sequence of events, or a particular ASID or 
task. For example, a sampling program that only accumulates data for a 
dispatch of a TCAM address space followed by a page fauh can be used to 
determine the value of further page fixing. 

Measure average I/O activity for each device and channel by examining digits 
6-8 of an SIO (type 0) or I/O (type 5) entry. I/O imbalances can thus be 
determined. (For MP configurations, each processor can be distinguished via 
CPU id). 



Example of Using the Trace Table 

The following example illustrates the mechanics of using the trace table. 

An examination of a trace table on a particular day showed what seemed to be 
an abnormally high number of program interrupts (entry type 3). Twenty-two 
out of 400 entries were PGM type entries. The elapsed time of the trace for the 
last wrap was calculated by subtracting the oldest entry time from the newest 
entry time. (This value is multiplied by 16 since the timer value is incremented 
only once every 16 microseconds.) For example, 

X'DE8041ED' - newest entry time 

- X'DESOODCE' - oldest entry time 

X' 34 IF' (*16) — elapsed time in microseconds = 

X'341F0' microseconds = 

213488 decimal microseconds = 

,213488 decimal seconds. 

The total number of program interrupts divided by the total elapsed time (in 
seconds) gives you the number of program interrupts per second: 

22 - number of PGM entries 

= 103 interrupts per second 

.213488 - elapsed time 

A different trace table was examined on this day. Computation revealed a 
similarly high rate of 97 interrupts per second. Calculating the rates from several 
traces of the previous day, however, showed values of only 20, 32, and 26 
interrupts per second. The rate had increased approximately 400% — a good 
indication that some change had been made that severely impacted the 
performance of the system. (Too many program interrupts indicate that the 
processor might be ineffectively utilized, or that it is being used to do a lot of 
unnecessary paging. Both are important factors in a performance evaluation.) 
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An investigation into the changes made to the system since the previous day 
revealed that one of the steal algorithm constants in IRARMCNS (the SRM table) 
had been altered. It was hoped that this change would improve performance; in 
fact, it degraded it. By resetting the constant, program interrupts returned to 
their previous rate and overall performance was back to its original level. 



Notes on Writing a Sampling I*rogram 

When writing a program to collect samples of the trace table, the following 
suggestions should be kept in mind: 

• If you are using the trace table to look at a sequence of events, you must copy 
the entire table into storage without being interrupted. Although you can 
usually move the trace table without interruption, a good technique for 
assuring this is to save the current entry time before moving the table. (The 
current entry is pointed to by the first word in the header, which is pointed 

to by location X'54'.) Then, after moving the table, compare the current entry 
time to the saved time. If the current entry time before and after the move is 
not the same, then the table was modified during the move. Throw the table 
away and start again. Do this sequence until the before and after times match; 
that is, until the table was not updated while being moved. 

• When moving the table, copy the oldest part first followed by the newest. 
This results in a table of events in time-sequence order. 

• You can copy the trace table into a larger area (a table with more than 32 
bytes per entry) to provide space for other calculations and information 
accumulated by your program. For example, it might be helpful to include in 
the entry the time between successive events or the jobname associated with 
each event. 

• Be sure you do not sample your own sampling program or a lot of useless data 
will be accumulated and analyzed. 

SYSl.LOGREC 

When gathering performance measurements, it is important that you also docu- 
ment the system environment that existed during the data collection period. A 
sophisticated analysis of seek activity on a DASD device is worthless unless you 
know what volume(s) was mounted on the device (information that is not pro- 
vided in GTF records, for example). The SYSl .LOGREC data set, which pro- 
vides a record of all hardware failures, selected software errors, and system condi- 
tions, is a valuable tool for this. It can also be useful as another source of 
performance data. Recoverable hardware and software errors that go unnoticed 
in a large complex might, in fact, be impacting overall system performance. 
Examples of performance-related data that can be recorded on the 
SYSl.LOGREC data set include the following: 

• CCH records — contain information for every channel failure that does not 
terminate the system. 

• DDR records - contain descriptions of operator and system swaps among the 
various devices. 

• MCH records — contain information on processor, storage, storage key, and 
timer failures. 
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• OBR records - contain information on paging I/O errors and I/O device 
failures. 

Each record in SYSl.LOGREC contains error statistics for the device (counts of 
the number of times that channels, machine models, and I/O devices have failed) 
and environmental data (the time and circumstances for each failure or system 
condition). Hardware statistics can also be accumulated for buffered log devices. 

The IFCEREPO service aid, which is used to retrieve the data from 
SYSl .LOGREC, reduces and formats the data as well. It enables you to examine 
the data in the form of edited reports, records, and/or record summaries such as 
the following: 

• the IPL report — contains comprehensive information on the reasons for 
repeated system initiaUzations, system idle time, and unavailability time. 

• the system error report - contains error statistics for hardware or software 
failures. 

The output from IFCEREPO should thus be considered an additional measure- 
ment tool when evaluating the performance of your system. 

For more information on the contents and format of the SYSl .LOGREC data 
set (and the IFCEREPO service aid program), see 0S/VS2 System Programming 
Library: SYSl.LOGREC Error Recording. 



Writing Data Reduction Programs 

Your installation will need data reduction programs to summarize and format the 
data provided by the MVS components (SMF and GTF), the system trace table, 
and the performance-related control blocks. Data that can be derived from the 
trace table and the control blocks are described in the preceding topic, "Other 
Sources of MVS Data." The SMF and GTF analysis programs, described in the 
"MVS Measurement Tools" section, are very useful reduction tools for the system 
components. These IBM-supplied programs summarize information from SMF 
and GTF that is often needed for a performance analysis. However, you might 
want to consider writing your own data reduction program in addition to, or in 
place of, any of those currently available. Your own program can measure 
variables that are pertinent to your particular installation's objectives and work- 
load and might, therefore, require unique types of calculations. 

This section provides general guidelines on the key measurements to be 
extracted from SMF and GTF in order to do a meaningful performance evalua- 
tion. Only a subset of the data collected by these measurement tools is per- 
formance-related; thus, your data reduction programs should minimize over- 
head and simplify interpretation by producing only that information having 
immediate potential for performance analysis. This section will help you 
determine how you can use the data that you might want to collect. This should 
be decided first so that irrelevant records are not produced. The following basic 
steps should then be used to develop your program: 

1 . Read the SMF/GTF record. 

2. Identify the type of record you are interested in via the record identifier or 
type fields. 
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3. Write selected records to a workfile. 

4. Sort and merge the selected records. 

5. Analyze the sorted data to detect excessive system wait time, inefficient use of 
I/O devices, or any other statistics that might lead to improved system 
performance. 

6. Format and print the selected data and summary statistics. 

Reducing SMF Data 

SMF produces the type of data that is needed to determine a job's resource 
consumption; that is, almost all the data collected is job-related. SMF data is 
crucial for jobstream tuning, which in turn is essential to the good performance of 
a systerti. An additional advantage is that certain SMF data can be summarized 
on a job class and performance group basis. SMF data reduction programs should 
be written to match MF/1 or RMF intervals to get a complete picture of system 
workload. (See OS/VSl System Programming Library: Initialization and Tuning 
Guide for examples of using these tools to compare an individual job's use of 
resources with that of the total system.) 

The following discussion illustrates how SMF data can be used to measure 
system resources. 



Measure where processor time is being spent. From records 5/35 and 4/34, obtain 
the processor time consumed by jobs and job steps. The processor time for each 
address space is broken down into execution time under TCBs and under SRBs. 
(See the topic "Categories of Processor Time" in chapter 11.2 for information on 
TCB and SRB time not reported by SMF.) Sort and summarize this data to 
determine which jobs are using large amounts of processor time. You can also 
focus on the specific elements of work that are using a lot of processor time by 
sorting the records by job or at the program level. 



Measure storage usage to identify potential shortages and inefficient use. Detailed 
information about how storage is being used by different address spaces can be 
obtained from records 4/34. These records provide the number of page-ins, page- 
outs, swap-ins, and swap-outs. Reduction of this data will indicate which jobs are 
doing most of the swapping and paging and at what frequency. 

From records 4/34, you can obtain the amount of virtual storage allocated and 
used by a job step or TSO session. These job step records can also be analyzed by 
your program to isolate the elapsed time and processor time used by job steps 
during the reporting period. The distribution of job steps in terms of storage used 
versus elapsed time or processor time can then be plotted. With this information, 
and the knowledge of how the jobs use the storage, you can identify the users and 
abusers of virtual storage. By translating the requests for virtual storage into their 
implications on real storage, and by knowing the storage size of the system, you 
will then be able to examine possible storage contention situations. A way to 
compute real storage from data in SMF records 4 and 34 is described in bulletin 
4.2.0 in chapter II.4, under the topic, "Separating Users with Large Working 
Sets." 
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SMF data reduction programs can be written to combine information from the 
4/34 records for job steps that overlap the same time interval. This data can be 
used to produce a report showing the storage utilization of the system at specific 
intervals. Specific periods of excessive storage waste and the jobs that caused it 
can thus be identified. This report will also indicate periods when there is low 
utiUzation of available storage. 



Measure I/O activity to determine the jobs that are using poor data management 
techniques (for example, blocking, MACRF, and RECFM). From records 14/15, 
obtain the following information in order to determine which data sets, if any, 
should increase their blocksizes, or otherwise improve their technique: 

• the data set name 

• the data set organization 

• EXCP counts issued against the data set 

• the blocksize of the data set 

• record format of the data set 

• macro form of the data set 

• the ddname 

• thejobname. 

A reduction program, while extracting the data, can apply selection criteria to 
further simplify the analysis. For example, those data sets with less than a 
specified EXCP count can be ignored in order to obtain information on high usage 
data sets only. Another valuable indicator of the I/O activity of a job is its SRB 
time, since most SRB time is accumulated by I/O interrupt processing. 



Measure system-wide resource activity. With a data reduction program, you can 
obtain from records 70-76 system-related data that was collected during an MF/1 
or RMF interval. Everything produced by MF/1 or RMF reports (processor, 
paging, workload, channel, and device activity) is provided in SMF records on an 
after-the-fact basis. For example, you can map the activity on all the channels 
and devices in use or only on specific ones to help determine where overloading 
situations have occurred. (For overall swap activity or swapping rates by per- 
formance groups, however, data contained in SMF records should be used with 
the rates provided by MF/1 or RMF; total swap counts produced by SMF can be 
misleading since swapping due to started or system tasks is not known to SMF.) 



Reducing GTF Data 

GTF, which traces system events, produces useful data for a performance analysis 
once it is reduced. In writing a data reduction program for GTF, remember that 
the format of GTF trace records depends on whether or not the record is time- 
stamped. This is indicated by a bit in the Timestamp Control Record. GTF 
record formats and contents are described in 0S/VS2 Service Aids Logic and 
0S/VS2 System Programming Library: Debugging Handbook. 
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Much of the data provided by GTF that is relevant to performance measure- 
ments can be extracted and summarized using the analysis programs already 
mentioned (see figures 1.4 - 1.7). When writing your own reduction program, keep 
in mind the types of data that are provided by these programs. They may suggest 
several methods of sorting and analyzing the available data in order to obtain 
some meaningful measurements. Because of the potentially high overhead of 
running GTF, be selective in the options you specify. 

The following discussion illustrates the major ways GTF data can be used to 
evaluate system resources. 



Measure the processor time that is being spent on supervisor activity. Via SVC 

trace records, GTF can identify the type, quantity, and users of supervisor 
services. Situations such as ENQ/DEQ bottlenecks can be determined by 
analyzing which services are in use, how they are being used, and by which jobs 
and modules. Focusing on the use of these services enables you to reduce 
processor bottlenecks, if necessary, by altering the time that an individual job 
spends on these services or by reducing the number of services that a job or jobs 
use during a period of time. 



Analyze program activity for packlists and BLDL lists. Using the data from GTF, 
a reduction program can order the modules on these lists by frequency of use. 
Packlists, or BLDL lists (for non-reentrant modules), can be recreated to allow for 
more efficient use of the processor and I/O resources. Analyzing the page faults 
of modules can also result in the creation of a new packlist. 



Measure I/O activity by device to obtain additional details. SIO and I/O trace 
records can provide information for each device such as the number of SIO 
operations, the extent of I/O interrupts, and the time intervals between SIOs. 
This can be very important when used with MF/1 or RMF. For example, the 
MF/1 or RMF Device Activity Report might show a particular DASD device busy 
75% of the time. By executing GTF and tracing the number of SIOs issued 
against the device, you can determine if there is a data set or VTOC placement 
problem. Since the SIO trace record contains the seek address from the DASD 
channel program, a reduction program can be used to illustrate graphically the 
nature of the seek activity on the busy volume. This graphic representation, when 
used with a LISTVTOC of the volume, can show that one or more data sets (or 
the VTOC) have been poorly placed on the volume. Based on this information, 
the data sets (or VTOC) can be rearranged on the pack for greater efficiency or 
can be redistributed to other, less active packs. 
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Monitor SYSEVENT activity during a selected time interval. SRM trace records 
track each SYSEVENT macro that is issued; that is, a record is written whenever 
the status of an address space or a system resource has changed. Use the data to 
analyze the effect on system performance of critical changes to the availability 
of resources such as real storage frames, auxiHary storage slots, and SQA virtual 
space. To help interpret the data, your reduction program can sort these records 
by address space ID, sysevent type, or time of occurrence for each job or TSO 
session. Note that sysevent type contains the TSO command or subcommand 
name in registers 1 5 and 1 . For further information on the use of GTF to track 
sysevents, and examples on reducing data from the SRM trace records, see 
0S/VS2 System Programming Library: Initialization and Tuning Guide. 



Monitoring Performance 



It is important to keep in mind that tuning your system is an on-going process. 
Measurement tools should be executed on a regular basis, not only when 
performance problems become apparent. You should be able, once your 
objectives are met, to identify potential performance problems before they 
become crises. 

Producing the same type of report or reports at periodic intervals should be a 
standard procedure at most installations. Data obtained in this way will enable 
you to determine the trends and the usual values for your particular installation. 
Once these values are established for individual workloads, you will be able to 
monitor performance. Any variation to the values usually observed can be 
indicative of a performance problem and warrants further investigation. 

A measurement tool such as RMF can be run continuously with little degrada- 
tion to the system's performance. Other tools, then, can be executed when RMF 
data indicates a potential problem area. The following are a few of the key 
measurements provided by RMF that should be examined regularly in order to 
monitor performance with maximum effectiveness: 

• CPU Activity Report — wait time percentage. 

• I/O Device Activity Reports — queue length and device busy percentages, 

(Note, however, that since RMF overhead is directly related to the number of 
device classes monitored, you should specify only those devices that require 
monitoring.) 

• Channel Activity Report - channel busy percentage (which provides feedback 
to the never-ending process of balancing DASD channel usage via data set and 
volume placement). 

• Paging Activity Report — system and address space total swapping rate 
(auxiliary and real storage shortage swaps indicate inadequate paging space or 
an abnormal amount of system page fix activity); main storage frame count 
(deviations from the norm, such as a growth in the size of SQA, can be easily 
identified). 

• Workload Activity Reports - all important data; especially useful for 
monitoring swapping and resource consumption on a performance group/ 
period/domain basis and for monitoring TSO response time. 
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• Page/Swap Data Set Activity — PLPA and common slot usage (confirms that 
the page data set configuration is as desired ; determines the page data sets 
where I/O errors are being encountered). 

• Trace Activity - SRM trace; domain tables. 

The SIR software monitor program is another possible tool for monitoring 
performance. It can assist you by identifying resource usage by individual jobs 
and address spaces. 
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Chapter 1.3 Pre-initialization MVS Performance Factors 



In addition to defining performane objectives and ensuring needed measurement 
tools are available, the following performance-related areas must be addressed in 
planning for and generating a performance -conscious MVS system: 

• the performance implications of the sysgen and initialization parameters you 
specify. The Initialization and Tuning Guide is the major source of informa- 
tion on this area. 

• basic performance factors, which are appHcable to all system control programs. 
This area was often ignored in early MVS systems and the following factors in 
particular should be carefully reviewed: 

— the adequacy of the hardware configuration. Insufficient hardware will 
affect the performance of any system and MVS's utiHzation of some hard- 
ware resources differs from OS/MVT and VS2 SVS. Review the adequacy 
of the configuration, especially if the system was not upgraded for MVS — 
see the topic "Configuration" in Part III. 

— I/O resource balancing. I/O bottlenecks have been a significant source of 
processor wait time in several existing MVS systems, yet this area of tuning 
is often ignored. Plan the configuration of I/O resources and data set place- 
ment carefully. Essentially, the rules that applied to MVT and SVS (such 
as placement of frequently used data sets near the VTOC) still apply to 
MVS. Chapter II. 3, "Investigating the Use of I/O Resources," in Part II 
describes measurements that can indicate possible I/O bottlenecks. Because 
of the dynamic nature of I/O requirements, I/O resource balancing is an on- 
going process and should be one of the major areas continuously reviewed as 
part of monitoring performance. 

"" Users that monopolize system resources. A common performance-related 
axiom is that "10% of the users use 90% of the system" and the installa- 
tion should identify and control these dominant users. If you isolate 
suspected dominant users in unique performance groups, RMF will report 
on the amounts of I/O, real storage, and processor service they use. SMF 
reports on virtual storage, processor, and I/O resources on a user level, but 
does not report on started tasks (you can also compute real storage on a 
user level from SMF data - see "Separating Users with Large Working Sets" 
in bulletin 4.2.0 in chapter II.4), SIR reports on started tasks, but includes 
little data on the use of I/O resources. Using RMF, SIR, SMF, or similar 
tools — or a combination of them — you can identify the dominant users at 
your installation. Control of dominant users depends on the user, its 
priority, and similar installation-dependent factors. For example: if more 
than one batch job uses significant amounts of real storage, you can assign 
them to the same job class and assign that class to a single initiator, so they 
do not execute concurrently; or use the rotate priority for a group of jobs 
with heavy processor demands. 

— Operational bottlenecks. Operational bottlenecks can be a significant 
source of system degradation but often are ignored because they are not 
usually reflected in system measurements. More information on operational 
bottlenecks is provided below. 
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Operational Bottlenecks 



In MVS, operators have a high degree of control over the system and their actions 
can seriously impact performance. For example, in MVT, an operator could start 
up to fifteen initiators; the number actually executed would depend on the 
amount of available storage. In MVS, if the operator starts fifteen initiators, all 
fifteen will attempt to run. 

Education of the operators and early, active involvement of the operations 
staff in MVS installation planning are keys to avoiding operational bottlenecks. 
Tuning operational characteristics of the system can be more difficult than 
tuning components because of the human factor — it is more difficult to tune 
human beings. Comments such as "I ran fifteen initiators on MVT" and "That's 
the way we did it on SVS" were common at some early MVS installations. 
Operators must understand basic MVS concepts and architecture in order to 
make intelligent decisions given a particular set of circumstances. Education of 
the operators is the most effective means of avoiding operational problems. 

Operational bottlenecks that do occur are not obvious and often are not 
reflected in system measurements. For example, delays due to volume mounting 
are not usually reported by measurement tools because these delays occur before 
the EXCP and subsequent SIO are issued; MVS measurement tools report pre- 
dominantly on the result of SIOs. Observation and awareness of common opera- 
tional bottlenecks are the keys to recognizing operational bottlenecks. Before 
doing detailed performance analysis, you should check for operational bottle- 
necks — if they exist, but are ignored, the analysis can be long and futile. The 
following paragraphs describe some operational problems observed in existing 
MVS systems. Note that solutions to operational bottlenecks are often simplistic, 
but not necessarily simple to implement. For example, if delays occur in volume 
mounting or responding to WTOR requests, an obvious solution is to mount 
volumes and reply quickly. But implementing this solution will probably require 
more than a suggestion or an edict to the operators. You might review the 
adequacy of the operations staff or the effectiveness of their procedures to 
determine the cause of the delay — for example, excessive paperwork required 
of operators might be causing delays. 



Number of Initiators 

The number of initiators to run on MVS is not directly related to the number to 
run on SVS or MVT because of differences in system architecture. And the 
number of MVS initiators is very dependent on the environment. The operations 
staff should not determine the number of initiators to run or the job classes to be 
assigned to initiators. Recommended values for different shifts and different 
workloads should be determined by the system programmers and communicated 
to the operators. The operators should also be educated on the reasons behind 
the recommended values; such education can help them withstand pressure from 
priority users complaining about their work. It has been observed that, when a 
user complains about a job, the operator often responds by starting another initi- 
ator for that job class — often resulting in further system degradation. 
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Forms Mounting 

Potential problems in this area include jammed or empty printers and operators 
not responding to output-related messages (FCB loads, paper changes, and so on). 
To help avoid operational bottlenecks in forms mounting, reconsider the criteria 
for special forms and use JES2 and JESS controls to group jobs with similar print 
requirements. 



Shift Changes 

At some installations, operators quiesce the system to finish work on one shift 
before switching to the next shift. This can result in significant waste of system 
resources. 



Input and Output 

Turnaround time reported by system measurements will not include delays 
entering work into the system and delays distributing output. Be especially 
suspicious of this problem occurring if complaints of slow turnaround time occur 
predominantly for areas under control of separate operational groups — for 
example, specifically for locally-entered and distributed jobs. 



Reply Delays 

In addition to the simplistic solution of ensuring that WTOR requests receive 
responses quickly, you might want to examine and re-code applications that 
issue excessive WTORs. Old applications, written for earlier systems, often 
depend unnecessarily on the operator (for example, to supply time-of-day 
information). 



Volume Mounting 

In JES2, only jobs in execution can cause volume mounting. In JESS, volume 
mounting can delay the job in setup or, if mounts are issued after MAIN 
SELECT, during execution. 

In addition to responding quickly to mount requests, you should consider 
controls for volume mounting. For example, you might examine the job mix 
and decide to run a large tape-spinning job on an off-shift. In JESS, the 
SDEPTH parameter might be too high for the number of initiators and tape 
drives in the system, resulting in tape drives being set up most of the time, 
although jobs using the drives have not been selected for execution. If a high- 
priority job enters the system, it might be necessary to restart setup to release 
needed tapes for higher-priority work. Watch for excessive restart setups, a 
long time spent on the main queue, a large number of jobs awaiting setup, 
and large fetch queue lengths. 
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PART II: Performance Analysis 



Chapter II. 1 : Steps in a Performance Problem Analysis 

Chapter II. 2: Investigating the Use of Processor Time 

Chapter II.3: Investigating the Use of I/O Resources 

Chapter II.4: Investigating the Use of Real Storage 

Chapter II.5 : User-oriented Performance Problem Analysis 
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Chapter II. 1 : Steps in a Performance Problem Analysis 



Figure II. 1 divides a performance problem analysis into five steps, each of which 
is described in more detail in the following paragraphs. Steps 1 and 2 are necessary 
preparation for a discipUned analysis; understanding the problem to be solved and 
gathering data to work with. Steps 3, 4, and 5 represent the hunt for the cause 
and solution to the problem. Each of these three steps requires more time, more 
effort, and/or more significant decisions than the preceding step: basic 
performance factors, applicable to all system control programs, are reviewed in 
step 3 ; the utihzation of system resources and specific potential bottlenecks are 
investigated in step 4 ; and solutions that are considered last resorts are included in 
step 5. The amount of time spent in each step depends on how quickly the 
problem is solved or how quickly the solutions investigated in a specific step are 
rejected. {Note: Carefully review steps 1 and 3; experience indicates that these 
steps are most often ignored in practice.) 



Step 


Cross-reference 


1 . Describe the performance problem in terms of 
the objectives that are not being met. 

A. user -oriented objectives 

B. system-oriented objectives 


Chapter 1.1 , "Defining Performance 
Objectives" 


2. Take a basic set of measurements. 


Chapter 1.2, "Selecting 
Measurement Tools" 


3. Review basic performance factors. 

A . hard wa re CO nf ig u rat io n 

B. I/O resource balancing 

C. control of users that monopolize resources 

D. operational bottlenecks 


Chapter 1.3, "Pre-initialization 
MVS Performance Factors" 


RE-MEASURE and RE-EVALUATE: 

Problem changed ? Go to step 1 . 
Problem solved ? Monitor performance. 




4. Identify bottlenecks in the system, contributors 
to the bottlenecks, and corrective actions. 

A. System-oriented problems— resource 
management 

1. Processor 

2. I/O resources 

3. Real storage 

B. User-oriented problems — workload 
management 


Chapter 11.2 
Chapter 11.3 
Chapter 11.4 

Chapter 11.5 


RE-MEASURE and RE-EVALUATE: 

Problem changed ? Go to step 1 . 
Problem solved ? Monitor Performance 




5. Investigate alternative solutions. 

A. modifying performance objectives 

B. obtaining more hardware 

C. modifying the system control program 





Figure II.l Steps in Performance Problem Analysis 

The following descriptions of the steps do not reflect their iterative nature. The 
process of re-measuring and re-evaluating (shown in figure II.l after steps 3 and 4) 
is necessary every time potential solutions are identified and implemented. The 
possible results: 
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The problem is solved and no new problems are identified; the analysis is 
completed. 

The immediate problem being addressed is solved but another problem become$ 
apparent. This is not unusual, because one bottleneck often disguises another. 
In this event, the analysis begins again with step 1 . 

The problem remains and further investigation is required; the analysis 
continues. 



STEP I . Describe the performance problem in terms pf the objectiveis that are npt 
being met. 

The purpose of this step is to be as specific as possible in describing the 
performance problem(s). The problem description should include the extent of the 
problem (which will help you predict the effectiveness of possible solutions) and 
the specific type of work experiencing the problem (which will help you identify 
possible trade-offs), For example, "Fifty percent, as opposed to the desired ninety 
percent, of TSO trivial transactions during peak hours are receiving three second 
response time" is a good problem description; "TSO is sluggish" is not. 

Problem statements will be specific if your performance objectives were specific: 
guidelines for defining performance objectives are included in Chapter I.l . Two 
types of objectives, based on two types of measurements, were defined: 

• user-oriented objectives - response time for a specific class of transactions or 
turnaround time for a specific application or class of jobs; and 

• system-oriented objectives — batch throughput, interactive transaction rate, or 
number of concurrent terminal users. 

Note which type of objective is not being met in your system — it will affect the 
approach taken to identify bottlenecks, as described in step 4. If both user- 
oriented objectives and system-oriented objectives are not being met, investigate 
the system-oriented problems first. 
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STEP 2. Take a basic set of measurements. 

The purpose of the basic set of measurements is to provide enough data to focus on 
specific potential problem areas in the system. "Enough" is defined by (1) the type 
of information you gather; and (2) the number of samples you take: 

(1) Measurements of the following should be included in the basic set of 
measurements: 

• system-wide resource utilization, to help identify those resources that are 
under- or over-utilized. RMF, MF/1, and hardware monitors provide such 
data. 

• resource utilization by job /address space. SMF reduction programs or 
system monitor programs such as SIR can be used to obtain detailed 
information on a job/address space level. 

In addition, GTF provides data on supervisor and I/O activities that is 
frequently needed for detailed performance analysis. Before beginning the 
analysis, many performance analysts execute GTF to trace at least SVCs and 
SRM events. If GTF is executed, request time-stamping to aid interpretation 
of the data. 

An overview of the tools mentioned here and of additional sources of data 
for possible reduction (such as control blocks and the trace table) is included 
in Chapter 1.2, "Selecting Measurement Tools." Use of additional tools, 
beyond those used to gather the basic set of measurements, can be dictated 
by the conclusions reached from reviewing the basic measurements. 

(2) It is important to have a multiple number of samples — for the times when 
performance problems are occurring and for the times when performance 
objectives are being met — and to document the workload at the time of 
each sample. This is necessary to judge if the reported value of any specific 
measurement indicates a possible problem area in the system. Recognition 
of values that might indicate a problem is usually based on the fact that a 
number is high or low compared to one of the following: 

• a value usually observed at the installation — for example, processor 
utilization observed under similar workload conditions or the average 
processor utilization over a period of time. If multiple samples have been 
gathered, or if measurement tools are executed on a regular basis, you will 
be able to identify usual values for your installation. 

• a target value obtained from experience-based guidelines, comparison with 
similar installations, or similar sources. This book provides target values 
for many of the measurements used in the suggested analysis. For 
example, several installations with satisfactory performance have observed 
a swap-to-TSO-transaction ratio of 1.1 :1 ; this can serve as a target value 
against which to judge swap-to-TSO-transaction ratios at your installation. 

In either case — usual or target values — the value should not become a goal. 
It should be used only to determine if further investigation of a measurement 
is indicated. Circumstances might have changed sufficiently for a "usual" 
value to no longer be valid and target values are not necessarily appropriate 
for your installation. In general, usual values are a more effective yardstick 
than target values. 
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Once a basic set of measurements has been taken, a major problem is determining 
which of those measurements to focus on. The analysis described in this Part of 
the Performance Notebook describes significant measurements to review in 
different situations. 



STEP 3. Review basic performance factors, which are applicable to all system 
control programs. 

The objective of this step is to ensure that basic tuning factors — those that are 
applicable to all system control programs — have been addressed before spending 
a lot of time and effort analyzing MVS specifically. The factors that should be 
addressed at this point have the following characteristics: 

• The effect (positive or negative) of varying the factor is fairly predictable. 

• The installation can easily exercise control over the factor — for example, via 
installation procedures or external system interfaces such as initialization 
parameters. 

Based on experience with existing MVS systems, the following factors are 
important to consider at this stage: 

• the adequacy of the hardware configuration in light of MVS system control 
program requirements. 

• I/O resource balancing, achieved by data set placement and the configuration of 
channels, control units, and devices. 

• control of users that monopolize system resources. 

• operational bottlenecks. 

These are the same factors that should be addressed prior to system generation and 
initialization — for details, see chapter 1.3, "Pre-initialization MVS Performance 
Factors;" the topic "Configuration-rules-of-thumb" in Part III includes guidelines 
against which you can review the adequacy of your configuration. 



STEP 4. Identify and correct bottlenedcs in the system. 

This step represents the body of the analysis. The approach for identifying and 
correcting bottlenecks depends on the type of performance problem being 
experienced - whether system-oriented objectives or user-oriented objectives are 
not being met (see step 1). There are two major areas of control in the system: 
resource management and workload management. Although they overlap, the 
initial focus is on resources for system-oriented problems and on workload manage- 
ment for user-oriented performance problems. 
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In the case of a system-oriented problem, the goal is to identify the resources 
that could be used more effectively and to improve their use. The resources of the 
system can be reduced to five basic resources: the processor, real storage, channels, 
control units, and devices. Bottlenecks that affect throughput will affect one 
(or more) of these resources and be reflected in measurements of its use. The initial 
focus of the analysis is determined by answering the question, "Is the system 
waiting ?" System wait time is defined as processor-idle time and is reported on 
the RMF or MF/1 CPU activity report: 

• If wait time is not high (for example, 10% or less of elapsed time), the analysis 
should focus on how processor time is being used, with the objective of reducing 
or eliminating non-productive processor time. This can lead to investigation of 
real storage, if significant processor time is being spent on paging or swapping; 
to investigation of I/O (with the intent of reducing EXCPs), if significant 
processor time is being spent on non-swap non-paging I/O interrupts; or to 
reduction of supervisor services such as SVCs. For details, see Chapter II.2, 
"Investigating the Use of Processor Time." 

• If wait time is high, you must determine why the system is waiting. Processor 
wait time is the result of four possible causes: (1) I/O delays; (2) excessive 
paging; (3) enqueues on serially reusable resources and internal locks; or 

(4) insufficient work in the system. Assuming the last is not the problem, the 
most significant areas to investigate are I/O delays and excessive paging; 
significant sources of enqueue contention are often data sets (for example, 
catalogs), and this contention contributes to I/O delays. For information on 
investigating the possibility of I/O delays, see Chapter II.3, "Investigating the 
Use of I/O Resources;" for the possible situation of paging causing wait time, 
see Chapter II.4, "Investigating the Use of Real Storage." 

Figure II.2 illustrates the logic of focusing on the major resources. 








I/O delays: 
investigate use of 
I/O resources 






^^ss^ Yes 


Chapter 11.3 


MF/1 or 




RMF CPU 
activity report 


Excessive paging: 
investigate use of 
real storage 







Chapter 11.4 



Investigate use of 
processor time 



Chapter 11.2 
Figure II.2 Focusing on the Major Resources when Investigating System-oriented Problems 
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In the case of a user-oriented problem, the analysis has already focused on a 
specific application or class of jobs or interactive transactions. The goal is to 
determine why the work is being delayed - to what resource(s) it is not receiving 
access — and to increase its access to the needed resource(s). Increasing access to 
resources by the work in question usually implies trade-offs for other work in the 
system (that is, workload management). If such trade-offs cannot be tolerated, the 
analysis must focus on optimizing use of the critical resource — that is, resource 
management. Chapter II.5 describes the investigation of specific user-oriented 
performance problems. 



STEP 5. If performance problems are not solved in step 4, review alternative 

solutions: modifying performance objectives; obtaining more hardware; 
or modifying the system control program. 

There is overlap between the steps of a performance analysis as outlined here. For 
example, any of these "alternative solutions" might have been identified in step 4 
as the most effective solution to a bottleneck; the question of additional hardware 
resources might have been addressed in step 3, in reviewing the reasonableness of 
the current hardware configuration. The key to exploring these "alternative 
solutions" at any point in the analysis is the ability to predict the probability of 
meeting your objectives by means of other solutions — workload balancing, 
increasing resource utilization, decreasing unproductive resource usage, and so on. 
The formulas for reviewing hardware reasonableness (see the topic "Configuration 
— rules-of-thumb" in Part III) might indicate the necessity of modifying your 
objectives or obtaining more hardware. Detailed performance objectives (as 
described in Chapter I.l) will allow you to identify minor changes to the objectives' 
priorities as solutions to resource bottlenecks identified in step 4. System 
modifications should be avoided. Before considering any system modification — 
or any solution — ensure it addresses a bottleneck identified in your system. There 
, are no panaceas among the performance suggestions, despite the reputation of 
some. Each solution must be considered in light of your installation and its specific 
bottlenecks. 
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Chapter II.2: Investigating the Use of Processor Time 



Investigation of processor time should be triggered by the presence of high 
processor time (for example, greater than 90% of elapsed time); if processor time is 
not high, the primary focus of the analysis should be on decreasing processor wait 
time (see the description of step 4 and figure II.2 in chapter II. 1 for information on 
the approach to performance analysis when processor wait time is high). There are 
essentially three ways to reduce the criticabess of the processor: 

1. Identify and reduce or eliminate non-productive processor time. 

2. If objectives still are not met and processor time is still high, focus on workload 
management: balancing the mix of processor- and I/0-bound jobs; rescheduling 
heavy processor users to less critical shifts. Note that, if you have defined 
detailed performance objectives (including resource requirements — see chapter 
I.l , "Defining Performance Objectives"), data will be available to help you 
determine effective workload changes. 

3. If objectives still are not met and processor time is still high, consider alternative 
solutions: modifying the performance objectives; or a larger or additional 
processor. 

This chapter describes the first approach: identifying and reducing or eliminating 
non-productive processor time. 



Identifying and Reducing Non-productive Processor Time 

Identifying and reducing non-productive processor time involves dividing processor 
time into categories, measuring how much time is used in each category, and 
identifying those categories that might be reduced or eliminated for a positive 
effect on performance without an adverse effect on system functions. 



Categories of Processor Time 

Two rules dictate the choice of meaningful categories into which to divide 
processor time: 

• The categories must be measurable — that is, the choice of categories depends 
on the measurement tools and reduction programs available at your installation. 

• You must know what activities contribute to the processor time in each 
category. 

Figure II.3 illustrates various divisions of processor time and the overlap of 
different categories. 
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(1 ) Total processor time 



(2) 



Time recorded by dispatcher 



Time not recorded 
by dispatcher 



(3) TCB time 



SRBtime 



other 



r 
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Problem program 



'front end^" of SVCs 



ASM 
SRB 



Post 

status 

SRB 



other 



RCT 
TCB 



other 



(5) Problem program Supervisor 



There are two types of SVCs: one is a single stage; the second includes two stages, the first 
initiating an action and the second processing the interrupt when the action is complete. 
"Front end" refers to the entire SVC, in single-stage SVCs; and to the first stage, in two-stage 
SVCs. For example, EXCP consists of two stages: the first to issue the SIO ; the second to 
process the interrupt. Issuing the SIO is the "front end" of EXCP. The second stage of two- 
stage SVCs is usually SRB time. 

F jgure 11. 3 Divisions of Processor Time 



The analysis described here focuses first on the TCB/SRB division (line 3 in 
figure II.3), and then on further division of TCB and non-TCB time (Une 4 in 
figure 11.3), depending on which seems excessive. To correctly interpret measure- 
ments of TCB and SRB time at your installation, you must know what TCB and 
SRB time is not reported by the measurement tools being used. For example, 
SMF, SIR, and RMF do not report different parts of TCB and SRB time: 

• RMF does not record TCB processor service units (from which TCB time can be 
computed, as described in bulletin 2.1 .0) for privileged address spaces (that is, 
the privileged bit on in the PPT to make the address space nonswappable unless 
it enters long wait). Instead of using the privileged bit in the PPT for such 
address spaces, consider placing them in a domain where the minimum MPL is 
greater than or equal to the number of address spaces in the domain. The address 
spaces will not be swapped and RMF will report their TCB time. 

• SMF does not report TCB or SRB time for started tasks (JES2, TCAM, the 
master scheduler). 

• With SIR, TCB and SRB time will be lost if the address space switches steps 
between samples. Time accumulated by the prior step after the sampling was 
taken is not reported. 

None of the tools report on time that is not recorded by the dispatcher. Time not 
recorded by the dispatcher includes: 

• I/O interrupt time 

• external interrupt time 

• page fault processing if not resolved by reclaim 

• PCFLIH time if valid SPIE in effect 

• processor "stopped" time if QUIESCE command used 

• time spent in active DSS breakpoints 
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• RCTtime 

• attention processing for TSO 

• page fault suspension processing 

• lock request suspension processing. 

Also, measurements of TCB and SRB time will be inaccurate if the operator 
stops the processor from the console by pressing the STOP key. The TOD clock 
will continue to run and the time will be reported under the state the processor was 
running when it was stopped. To avoid this, use the QUIESCE command to stop 
the processor. 



Focusing on Categories of Processor Time 

Figure II.4 illustrates the logic of focusing on different categories of processor time 
and includes cross-references to bulletins at the end of this chapter, which describe 
the investigation of specific categories. 

The direction of the investigation is indicated by the proportion of processor 
time that is TCB time. The judgment of TCB time as "high" or "low" is 
installation-dependent and is best made in light of the proportion of TCB time 
usually observed at your installation. As a starting point for new installations, a 
target value of 50% (as measured by RMF, assuming that no address spaces are 
marked privileged in the PPT) might be used : if TCB time is greater than 50%, 
investigate it; if less than 50%, determine the major areas of use of non-TCB time. 



C 
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Bulletin 2.1.0 
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Analyze 
SVCs 



Bulletin 2.3.0 



Identify heavy 

processor 
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Determine major areas 
of use of non-TCB time: 

• swapping 

• non-swap paging 

• I/O interrupts 



Bulletin 2.2.0 



Figure II.4 Focusing on Categories of Processor Time 
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Note that SRB time itself is not included in figure II.4. ("Non-TCB time" 
includes reported SRB time and unreported processor time ; much of the processor 
time used for swapping, paging, and I/O interrupts is unreported time.) SRB time 
is accumulated for services such as POST STATUS, cross-memory POST, cross- 
memory STATUS, the timing SRB, and the SRM timer pop SRB. Although it is 
possible to roughly determine hojv much SRB time is being spent by the various 
services (using GTF and a nucleus map), it is not necessarily fruitful. It is not 
usually possible to reduce the processor time spent by most of these services: the 
effort of breaking down SRB time is not clearly justified by the possibility of the 
gain. 

Additional bulletins on analyzing the use of processor time will be added as 
information becomes available. 
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Computing TCB Time 

Processor service units are accumulated only for TCB time. As a result, TCB time 
can be computed via the following formulas: 

total CPU service units 



TCB seconds = 



CPU coefficient x maximum CPU service units per second 



TCB seconds 

and — -— ; — X 100 = TCB % 

RMF mterval seconds 

where: 

• total CPU service units is reported on the RMF workload activity report. 
(Service units from the MF/1 workload activity report can be used if the CPU 
coefficient is the only non-zero coefficient.) 

• the CPU service definition coefficient was specified by the installation in the 
IPS and is reported on the RMF workload activity report. (Note : Do not 
confuse the system tuning parameters on the left-hand side of the report with 
the service definition coefficients on the right-hand side.) 

• maximum CPU service units per second are listed in figure II. 5. The values 
hsted are internal SRM constants. 

All TCB time recorded by the dispatcher is reported by RMF (and therefore wiU be 
included in this computation) except for privileged address spaces (the privileged 
bit turned on in the PPT) and TCB time between jobs (that is, initiator TCB time 
when a job is not running). To capture TCB time for "privileged" address spaces, 
place them in a domain where the minimum MPL is greater than or equal to the 
number of address spaces in the domain, instead of using the PPT. TCB time 
between jobs is not usually a significant factor. 

Non-TCB time, then, is: 

100 - (CPU wait time % + TCB %) = non-TCB % 

where CPU wait time is reported on the RMF or MF/1 CPU activity report. 
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Processor Analysis 

• computing TCB time 
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Computing TCB Time (continued) 



March 1977 



Note that this formula can also be used to determine TCB time for any of the 
distinct breakdowns of work reported in the RMF workload reports — performance 
groups, performance objectives, domains — by using the CPU service units for that 
group of work. For example, SMF does not report on started-task address spaces. 
You can use RMF to determine TCB time for started-task address spaces by 
assigning them to a separate performance group. 



Processor Model 


Maximum CPU service units per second 




(assuming CPU coefficient of 1) 
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Figure 11.5 Maximum CPU Service Units pet Second 
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Breakdown of non-TCB time 

Most processor time other than TCB time recorded by the dispatcher is due to 
swapping, I/O interrupts, and non-swap paging. The purpose of this bulletin is to 
help you determine the approximate percentage of processor time used for 
swapping, I/O interrupts, and paging — thereby identifying the area where further 
investigation might provide most significant performance improvement. 

Figure II.6 summarizes the formulas presented in this bulletin. The formulas for 
computing processor time for swapping, non-swap paging, and I/O interrupts are 
derived from the following formula for total non-TCB time, based on a 3.7 TSO 
and batch system with SUs 1-7 installed: 

n-^„„, [5i,000(SR) + 2,000(SPR) + 4,000(NSPR) + 2,000(SIOR)] xC .^f. 
non-TCB %« tx luu 

where : 

• SR is the swap rate — swaps per second, as reported in the swap sequence counts 
on the RMF paging activity report. 51,000 is the approximate processor time, in 
microseconds, for RCT (region control task) processing on a 1 58-3 UP. 

• SPR is the swap paging rate — swap page-ins plus swap page-outs divided by the 
RMF interval in seconds, as reported on the paging activity report. 2,000 is the 
approximate processor time, in microseconds, for ASM (auxiliary storage manager) 
processing on a 1 58-3 UP, not including approximately 2,000 microseconds to 
process each I/O interrupt. 

• NSPR is the total system non-swap paging rate - non-swap page-ins plus non-swap 
page-outs (including VIO, non-VIO, system areas, and private areas), divided by 
the RMF interval in seconds, as reported on the paging activity report. 4,000 is 
the approximate processor time, in microseconds, for ASM processing on a 
158-3 UP, not including approximately 2,000 microseconds to process each I/O 
interrupt. 

• SIOR is the SIO rate — total channel activity counts divided by the RMF interval 
in seconds, as reported by the channel activity report. 2,000 is the approximate 
time, in microseconds, for I/0-interrupt processing on a 158-3 UP. 

• C is a coefficient that reflects the approximate relative time to perform system- 
control-program (SCP) functions on various processors, as normalized to the 
158-3 UP. (Therefore, for the 158-3 UP, C is 1.0.) Figure II.7 provides approxi- 
mate coefficients for use in this formula. 
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A. Basic formula for non-TCB % 

non-TCB to/o 



[51,000{SR) + 2,000(SPR) + 4,000(NSPR) + 2,000(SIOR)] x C ,„„ 

%W ' X 100 

10^ 



To check fornnula for non-TCB %: 

result of formula A = non-TCB % from RMF (bulletin 2.1 .0) (within « 5%) 



B. Formula for swap non-TCB % 

[51,000(SR) + 2,000(SPR) + 2,000(SPR -^ 3)] x C 



swap non-TCB %5^ 



assuming 3 pages per SIO. 



X 100 



10^ 



C. Formula for non-swap paging non-TCB % 

-rno 0/ ^ [4,000(NSPR) + 2,000(NSPR f 2)] x C , .„ 
non-swap paging non-TCB % W — ■ • x 100 



10"= 



assuming 2 pages per SIO. 



D. Formula for non-swap non-paging I/O interrupt non-TCB % 

non-swap non-paging I/O interrupt non-TCB %f^ 

[2,000(SIOR - (SPR ^ 3) - (NSPR f 2))] x C 



X 100 



10" 



which excludes I/O interrupts for non-swap paging and swapping, assuming 3 pages 
per SIO for swapping and 2 pages per SIO for non-swap paging. 



Legend: SR = swap rate 

SPR = swap page rate 

NSPR = non-swap page rate 

SIOR = SIO rate 



Figure II.6 Fonnulas for Breakdown of non-TCB Processor Time 

From experience to-date, this formula is accurate within approximately 5%. 
However, the times for RCT, ASM, and I/0-interrupt processing on a 158-3 UP, 
and the coefficients reflecting the relative time to process these functions on 
different processors, are approximations and can vary significantly from the 
values given. As a result, it is imperative to check the result of the preceding 
formula. Compare the result of this formula to the non-TCB percentage 
computed via RMF (assuming no address spaces are marked privileged in the 
PPT, so that all TCB time recorded by the dispatcher is reflected in RMF 
measurements — see Bulletin 2.1 .0). If the results differ by more than 5%, the 
fault might lie v^dth an unusual activity in your system that is consuming 
processor time. Review system activity at the time of the report. For example, 
in one case where this formula seemed incorrect, an excessive number of non- 
swappable address spaces were active in the system. The dispatcher was 
"tripping" over them on the dispatching queue and dispatcher processor time 
had increased dramatically. 
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Processor Mode! 


Approximate Coefficient 


Note: These coefficients are approximations, based on measurements of svstem- 
control-program (SCP) functions in certain jobstreams that reflect common user 
environments, and normalized to a 1 58-3 UP; the jobstreams were executed on 
an MVS release 3.7 system with SUs 1-7 installed. The coefficients do vary and, 
therefore, results of any formulas using these coefficients must also be con- 
sidered approximations. These approximate coefficients apply only to SCP 
functions; they should not be used for comparing non-SCP programs on 
different processors. 


145 
155 

158 UP 
158 MP 
158-3 UP 
158-3 AP/MP 

165 

168 UP 8K cache 

168 UP 16K cache 

168 half-duplex 16K cache 

168 MP 

168-3 UP 

168-3 half -duplex 

168-3 AP/MP 


2.60 
1.46 

1.11 
.67 

1.00 
.59 

.52 

.47 
.42 
.43 
.274 

.36 
.37 
.213 



Figure 11.7 Coefficients Reflecting Relative Time to Perform SCP Functions on 
Different Processor Models 

If the result of this formula seems correct (that is, within approximately 5% of 
other computations of non-TCB time), the following formulas can be used to 
compute approximate non-TCB time used for swapping, non-swap paging, and I/O 
interrupts: 



• swapping non-TCB % « 

[51,000(SR) + 2,000(SPR) + 2,000(SPR - 3)] x C 
10^ 



xlOO 



where (SPR-?- 3) reflects the number of SIOs, assuming three pages for each 
SIO.^ The assumption of 3 pages per SIO is based on a working set size of 12 
pages, five of which are LSQA; and four local paging data sets on 3330 devices. 
The number of pages per SIO will tend to increase if: the working set is larger; 
page data sets are on a drum; the swap load is very heavy (it is more likely that 
as many pages as possible will be swapped in/out in each ASM burst); or the 
ASMBURST value is raised by the installation. The number of pages per SIO 
will tend to decrease if more local page data sets are added or if the ASMBURST 
value is lowered by the installation. 



-'^if all paging data sets are dedicated, the average number of pages per SIO for swapping and 
non'swap paging combined is: 

total paging rate 



sum of paging devices' activity counts per second 
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Breakdown of non-TCB time (continued) 

If swapping processor time is judged excessive, the solution is to reduce 
swapping — see bulletin 4.1 .0 in the chapter, "Investigating the Use of Real 
Storage." 



• non-swap paging non-TCB % « 

[4,000(NSPR) + 2,000(NSPR 4- 2)] x C 
10® 



X 100 



where (NSPR -f- 2) reflects the number of SIOs, assuming two pages per SIO.^ 
The number of non-swap pages per SIO varies with the amount of VIO and 
demand paging. As VIO and demand paging increase, the number of pages per 
SIO increases up to the maximum that can be handled in a single ASM burst 
(a probable maximum of 5 on a 3330 and 10 on a 2305 during a 50-millisecond 
burst). The assumption of 2 pages per SIO is based on the paging data sets being 
on 3330s. 

If non-swap paging processor time is judged excessive, compare non-VIO and 
VIO paging rates to determine where decreases will potentially have the more 
significant effect. Bulletins 4.2.0 and 4.3.0 in chapter II.4 describe decreasing 
demand paging and VIO paging, respectively. 



• non-swap non-paging I/O interrupt non-TCB % i 
2,000[(SIOR - (SPR -f- 3) - (NSPRf 2))] x C 
10® 



x 100 



This formula excludes processor time used for I/O interrupts for non-swap 
paging and swapping (based on the preceding assumptions of 3 pages per SIO 
for swapping and 2 pages per SIO for non-swap paging). If I/O interrupt 
processor time is judged excessive, the solution is to reduce the number of 
EXCPs/SIOs - see bulletin 3.4.0 in the chapter II.3, "Investigating the Use of 
I/O Resources." 

Figure II.8 is an example of applying the formulas presented in this bulletin. 



^ If all paging data sets are dedicated, the average number of pages per SIO for swapping and 
non-swap paging combined is: 

total paging rate 



sum of paging devices' activity counts per second 
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System: 158-3 UP 3 meg MVS 3.7 with SUs 1-7 and RMF applied. 
TSO and batch 



Computing TCB time via RMF: 

Data 

CPU busy = 100-3.14 = 96.86% 

total CPU service units = 1 ,057,298 

CPU coefficient = 10.0 

maximum CPU service units/second = 51 .2 

interval = 60.12.862 = 3612.86 seconds 



Source 

CPU activity report 

workload activity report 

workload activity report 

figure 11.5 

workload activity report 



TCB seconds 



1,057,298 
51.2 X 10 



= 2065.04 



TCB% =-?2^££± X 100 = 57.16% 
3612.86 

non-TCB % = 96.86-57.16 = 39.70% 



Formula for non-TCB time: 

Data 

SR = 1.49 

SPR = 23.91 + 21.46 = 45.37 

NSPR = 16.15 + 3.60 = 19.75 

SIOR = 323,492-^-3612.86 = 89.54 

C= 1.0 



Source 

RMF paging activity report 
RMF paging activity report 
RMF paging activity report 
RMF channel activity report 
Figure 11.7 



non-TCB % 



,[51,000(1.49) + 2,000(45.37) + 4,000(19.75) + 2,000(89.54)] x 1.0 



xlOO 



10^ 



142.48% 



Testing formula for non-TCB time: 

42.48 - 39.70 = 2.78% (within 5%) 



Formula for swapping non-TCB %: 

-r^o 0/ ^[51,000(1.49) + 3,000(45.37) + 2,000(45.37 -^ 3)] X 1.0 _^ 
swapping non-TCB % ^^ — ■ = — '^x 100 



10" 



119.70% 



Formula for non-swap paging non-TCB %: 



non-swap paging non-TCB % 



[4,000(19.75) + 2.000(19.75 ^2)] x 1.0 



X 100 



10" 



19.88% 



Formula for non-swap non-paging I/O interrupt non-TCB %: 

[2,000(89.54 - (45.37 -^ 3) - (19,75 4-2))] x 1.0 



I/O interrupt non-TCB % 



X 100 



10^ 



! 12.91% 



Figure II.8 Example of Applying Formulas for Breakdown of non-TCB Time 
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SVC analysis 

The purpose of analyzing SVCs is to identify those SVCs that are *'costly" in terms 
of processor time and whose "cost" can be decreased — by decreasing the 
frequency of the SVC or by reducing the SVCs processor time. The factors 
important in identifying such SVCs are: 

• Recognition of SVCs whose operation or frequency can be changed without 
having an adverse effect on system functions. The topic "SVCs" in Part III 
presents suggestions on reducing the frequency or processor usage of some SVCs. 

• The frequency of SVCs, available by trace table or GTF analysis, GTFSVC, 
lUP # 5796-PGE, is a GTF reduction program that provides SVC frequency on a 
system and job level. To use frequency to focus on specific SVCs, you should 
have a fairly clear idea of SVC frequencies usually observed in your system when 
performance objectives are being met. SVCs cannot be judged only by relative 
frequency, since relative frequency does not equate to relative processor cost; 
ten calls of one SVC can equal, in terms of processor time, one call of another, 
However, since it is not simple to determine relative processor cost of the SVCs 
(which is the third factor in SVC analysis), frequency is often used as an inexact 
indicator. 

• The processor time used by SVCs. Although the process is involved, this can be 
computed using GTF or the trace table. The following paragraphs describe a 
method for doing this. 



Computing Relative Processor Cost of SVCs 

Computing the processor cost of SVCs is based on measuring the TCB seconds 
spent executing each SVC, which is not a trivial task. To determine the TCB 
time for each SVC, use GTF or trace table data to obtain: 



The total TCB time spent executing each SVC (all calls of the SVC). Each time 
an SVC is issued by each task in the system, take the difference from the time 
the SVC started to redispatch of the task. If the task is interrupted, the time 
between the interrupt and reinstatement of the task must be subtracted from the 
SVC execution time. Note also that an SVC can call one or more additional 
SVCs. The time spent executing the "nested" SVCs should be included in the 
processor time of the base SVC, although you might also want to report TCB 
time for the nested SVCs individu^ly. 
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SVC analysis (continued) 

2. The average time spent executing eacli call of the SVC, by dividing the total 
TCB time spent executing the SVC by the frequency of that SVC in the sample 
period. The TCB time of each call of an SVC will vary; in some cases, this 
variance will be significant enough that the average TCB seconds for that SVC 
might be meaningless, unless a great number of samples are used. 

(Note that a less exact, but simpler, way of obtaining relative size of SVCs is to 
track the elapsed time of each SVC — from the time the SVC is issued until 
redispatch of the task, which does not take into account factors such as 
interruption of the task.) Once the average TCB time for each SVC is known 
for your system, you can determine the processor "cost" of each SVC for 
any given measurement interval as a percentage of total processor utiUzation 
or as a percentage of total TCB time: 



SVC A's TCB % of processor utilization « 

avg. TCB seconds for SVC A x frequency per second of SVC A 
processor utilization 



xlOO 



SVC A's % of total TCB time « 

avg. TCB seconds for SVC A x frequency per second of SVC A ^qq 
total TCB % 

where frequency per second for the period of time being investigated can be 
determined by GTF analysis, trace table analysis, or obtained from GTFSVC 
(lUP #5796-PGE); and processor utilization or total TCB % is expressed as a 
fraction. 

Note that, if your installation includes more than one type of processor, 

you can use the coefficients in figure 11,7 (included in bulletin 2.2.0) to 

convert SVC time measured on one processor to the approximate SVC time 

on a different processor: 

Co X TCB seconds for SVC A 
TCB seconds for SVC A ^ — 

where Cj is the coefficient of the processor on which the SVCs were 
measured and C2 is the coefficient of the processor on which you are 
projecting SVC time. 
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SVC analysis (continued) 

Figure II.9 is an example of average TCB processor microseconds measured for 
SVCs on a 1 68-3 half duplex (HD) 6 meg TSO and batch system (approximately 
90% TSO, 10% batch) with processor utiUzation of 88%, computed by means of 
trace table reduction. The last column in figure II.9 illustrates converting the 
average times measured on the 168-3 HD to the approximate average times the 
SVCs would take on a 158-3 UP. (Some performance analysts find the 158-3 UP 
base numbers to be the most convenient form in which to use the values.) The 
formula used for the conversion, using the coefficients for the 158-3 UP and the 
168-3 HD from figure II.7, is: 

average TCB seconds for SVC A on 158-3 UP w 
1.0 X avg. TCB seconds for SVC A on 168-3 HD 
.37 

Figure II. 9 illustrates the possible wide variance in processor time spent by SVCs, 
in relation to their frequency. For example, computing TCB % of processor 
utilization for SVCs (EXCP/XDAP) and 10 (GETMAIN/FREEMAIN) on the 
168-3 HD (where the microseconds, taken from figure II.9, have been converted 
to seconds and processor utilization was 88%): 

• SVC TCB % = -000788 x 93.4 j^ ^ qO = 8.36% 

.88 

• SVC 10 TCB % = 000238 x 232.2 ^ 100 = 6.28% 

.88 

Although SVC 10 was executed two-and-a-half times more frequently than SVC 
(232.2 versus 93.4 calls per second), SVC accounted for 33% more TCB pro- 
cessor time. Also, SVC is a two-stage SVC and the TCB time is only the "front 
end;" approximately the same amount of processor time (SRB time) is spent in 
the second stage of the SVC, fielding the interrupt. 

Figure II.9 is included only as an example. The average TCB microseconds listed 
can vary significantly from those at your installation. 
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Warning: This figure is included only as an example. The average TCB time spent executing an SVC depends 






significantly on the characteristics of the workload being supported; the times listed in this figure can .vary 






significantly from those at your own installation. Also note that: 






• the lower the frequency of the SVC, the greater the possible distortion in its computed average TCB time. 






• the observed frequency includes all SVCs that were issued during the time of the trace table sample; however, 






computation of average TCB time is based only on SVCs that were issued and completed within the time of 






the trace table sample. 






System Level: 1 68-3 half-duplex (HD) 6 meg TSO ( w 90%) and batch ( w 1 0%) 






MVS Release 3.7 with RMF, RACF, TSO Command Package, NJE FaciHty for JES2, 






and the following SUs installed: 1 , 2, 4, 5, 6, 7, 8, 10, 15 














Average TCB Microseconds 














(including TCB time of other 












Average TCB Microseconds 


SVCs called by base SVC) 










Observed Frequency 


(including TCB time of other 


on 158-3 UP, Converted 










(per second) on 


SVCs called by base SVC) 


from Times Measured on 






SVC No. 


SVC Name 


168-3 HD 


Measured on 168-3 HD 


168-3 HD 









EXCP/XDAP 


93.4 


788 


2130 






1 


WAIT/WAITR 


93.0 


199 


538 






2 


POST/PRTOV 


5.1 


182 


492 






3 


EXIT 


15.8 


(see note 1 ) 








4 


GETMAIN 


6.3 


343 


927 






5 


FREEMAIN 


.7 


431 


1165 






6 


LINK 


1.9 


(see note 2) 








7 


XCTL 


4.1 


376 


1016 






8 


LOAD 


7.5 


490 


1324 






9 


DELETE 


14.0 


250 


676 






10 


GETMAIN/FREEMAIN 


232.2 


238 


643 






11 


TIME 


7.3 


122 


330 






12 


SYNCH 


4.3 


(see note 2) 








13 


ABEND/(EOT) 





9200* 


24865 






14 


SPIE 





943* 


2549 






15 


ERREXCP 


.1 


176 


476 






16 


PURGE 


3.7 


6786 


18341 






17 


RESTORE 













18 


BLDL/FIND 


4.1 


2739 (see note 3) 


7403 (see note 3) 






19 


OPEN 


.7 


10710 


28946 






20 


CLOSE 


1.7 


1709 


4619 






21 


STOW 













22 


OPEN (TYPE=J) 


.3 


8335 


22527 






23 


CLOSE (TYPE=T) 


.1 


9638 


26049 






24 


DEVTYPE 


.1 


303 


819 






25 


TRKBAL 


.1 


336 


908 






26 


CATALOG/INDEX/LOCATE 


1.3 


10403* 


28116 






27 


OBTAIN 


.3 


4357* 


11776 






*These niimhers wprp obtained from a different sample* in this sample -t^Te-TGB time was not eaptttfed-or^HTe'"fretttrency~of 






the SVC was zero. 






Notes: 






1 . EXIT does not return to the next sequential instruction; EXIT is included in the SVC that issues the EXIT. 






2. According to the technique used, time spent executing the linked or synched program would have been counted as part of 






this SVC. As a result, no time was computed for LINK or SYNCH. 






3. The processor time of BLDL varies widely according to the amount of I/O done. 





Figure II.9 Example of Average TCB Times for SVCs on a 168-3 6 Meg TSO and Batch System (Part 1 of 4) 
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Warning: This figure is included only as an example. The average TCB time spent executing an SVC depends 


significantly on the characteristics of the workload being supported; the times listed in this figure can vary 


significantly from those at your own installation. Also note that: 


• the lower the frequency of the SVC, the greater the possible distortion in its computed average TCB time. 


• the observed frequency includes all SVCs that were issued during the time of the trace table sample; however, 


computation of average TCB time is based only on SVCs that were issued and completed within the time of 


the trace table sample. 


System Level: 1 68-3 half-duplex (HD) 6 meg TSO ( « 90%) and batch ( « 1 0%) 


MVS Release 3.7 with RMF, RACF, TSO Command Package, NJE Facility for JES2, 


and the following SUs installed: 1, 2, 4, 5, 6, 7, 8, 10, 15 










Average TCB Microseconds 










(Including TCB time of other 








Average TCB Microseconds 


SVCs called by base SVC) 






Observed Frequency 


(including TCB time of other 


on 158-3 UP, Converted 






(per second) on 


SVCs called by base SVC) 


from Times Measured on 


SVC No. 


SVC Name 


168-3 HD 


Measured on 168-3 HD 


168-3 HD 


28 


CVOL 









29 


SCRATCH 


.1 


1902* 


5141 


30 


RENAME 









31 


FEOV 









32 


[no name] 


.1 


1324* 


3578 


33 


lOHALT 









34 


MGCR/QEDIT 


.1 


2941 


7949 


35 


WTO/WTOR 


.3 


3612 


9762 


36 


WTL 


.3 


1686 


4557 


37 


SEGLD/SEGWT 









38 


[reserved] 








39 


LABEL 









40 


EXTRACT 


2.5 


130 


351 


41 


IDENTIFY 


.1 


431 


1165 


42 


ATTACH 


.3 


919 


2484 


43 


CIRB 









44 


CHAP 









45 


OVLYBRCH 


.7 


5435 


14689 


46 


TTIMER 









47 


STIMER 


.7 


827 


2235 


48 


DEQ 


23.8 


290 


784 


49 


[reserved] 








50 


[reserved] 








51 


SNAP/SDUMP 









52 


RESTART 









53 


RELEX 









54 


DISABLE 









55 


EOV 


1.5 


2155 


5824 


56 


ENQ/RESERVE 


26 


336 


908 


57 


FREEDBUF 









58 


RELBUF/REQBUF 









59 


OLTEP 









60 


STAE/STAI/ESTAE/ESTAI 


40.6 


123 


332 


61 


IKJEGS6A 









62 


DETACH 


.5 


213 


576 


63 


CHKPT 









64 


RDJFCB 









65 


[reserved] 








*These numbers were obtained from a different sample; in this sample, the TCB time was not captured or the frequency 


of the SVC was zero. 



Figure II.9 Example of Average TCB Times for SVCs on a 168-3 6 Meg TSO and Batch System (Part 2 of 4) 
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Warning: This figure is included only as an example. The average TCB time spent executing an SVC depends 


significantly on the characteristics of the workload being supported; the times listed in this figure can vary 


significantly from those at your own installation. Also note that: 


• the lower the frequency of the SVC, the greater the possible distortion in its computed average TCB time. 


• the observed frequency includes all SVCs that were issued during the time of the trace table sample; however, 


computation of average TCB time is based only on SVCs that were issued and completed within the time of 


the trace table sample. 


System Level: 1 68-3 half-duplex (HD) 6 meg TSO ( « 90%) and batch ( « 1 0%) 


MVS Release 3.7 with RMF, RACF, TSO Command Package, NJE Facility for JES2, 


and the following SUs installed: 1, 2, 4, 5, 6, 7, 8, 10, 15 










Average TCB Microseconds 










(including TCB time of other 








Average TCB Microseconds 


SVCs called by base SVC) 






Observed Frequency 


(including TCB time of other 


on 158-3 UP, Converted 






(per second) on 


SVCs called by base SVC) 


from Times Measured on 


SVC No. 


SVC Name 


168-3 HD 


Measured on 168-3 HD 


168-3 HD 


66 


BTAMTEST 









67 










68 


SYNADAF/SYNADRLS 









69 


BSP 









70 


GSERV 









71 


ASGNBFR/BUFINQ/RLSEBFR 









72 


CHATR 


2.9 


1413 


3819 


73 


SPAR 









74 


DAR 









75 


DQUEUE 









76 


[no name] 









77 


[reserved] 








78 


WSCAN 









79 


STATUS 


25.4 


141 


381 


80 


[reserved] 








81 


SETPRT 









82 


DASDR 









83 


SMFWTM 


3.1 


4494 


12146 


84 


GRAPHICS 









85 


DDRSWAP 









86 


ATLAS 









87 


DOM 









88 


MOD88 









89 


[reserved] 








90 


[reserved] 








91 


VOLSTAT 









92 


TCBEXCP 









93 


TGET/TPUT 


3.3 


687 


1857 


94 


STCC 


.1 


400 


1081 


95 


SYSEVENT 


8.1 


260 


703 


96 


STAX 


.9 


137 


370 


97 


TEST -TSO 


.1 


495 


1338 


98 


PROTECT 









99 


DDDYNAM 


2.1 


11476 


31016 


100 


IKJEFFIB 









101 


QTIP 


4.7 


408 


1103 


102 


AQCTL 









103 


XLATE 










Figure II.9 Example of Average TCB Times for SVCs on a 168-3 6 Meg TSO and Batch System (Part 3 of 4) 



11.26 0S/VS2 MVS Performance Notebook 



Bulletin 2.2.0 

SVC analysis (continued) 



Processor Analysis 
• SVC analysis 

March 1977 



Warning: This figure is included only as an example. The average TCB time spent executing an SVC depends 
significantly on the characteristics of the workload being supported; the times listed in this figure can vary 
significantly from those at your own installation. Also note that: 

• the lower the frequency of the SVC, the greater the possible distortion in its computed average TCB time. 

• the observed frequency includes all SVCs that were issued during the time of the trace table sample; however, 
computation of average TCB time is based only on SVCs that were issued and completed within the time of 
the trace table sample. 

System Level: 1 68-3 half-duplex (HD) 6 meg TSO ( « 90%) and batch ( « 1 0%) 

MVS Release 3.7 with RIVIF, RACF, TSO Command Package, NJE Facility for JES2, 
and the following SUs installed: 1 , 2, 4, 5, 6, 7, 8, 10, 15 


SVC No. 


SVC Name 


Observed Frequency 
(per second) on 
168-3 HD 


Average TCB Microseconds 
(including TCB time of other 
SVCs called by base SVC) 
Measured on 168-3 HD 


Average TCB Microseconds 
(including TCB time of other 
SVCs called by base SVC) 
on 158-3 UP, Converted 
from Times Measured on 
168-3 HD 


104 
105 
106 
107 
108 
109 
110 
111 
112 
113 

114 
115 
116 
117 
118 
119 
120 
121 
122 
123 
124 
125 
126 
130 
131 
132 
133 


TOPCTL 
IMAGLIB 

MODESET 

[reserved] 

ESR (type 3 & 4) 

DSTATUS 

[calls HASPSSSM] 

PGRLSE 

PGFIX/PGFREE/PGLOAD/ 

PGOUT 
EXCPVR 
[reserved] 
ESR (typel) 
DEBCHK 
[reserved] 
TESTAUTH 
GETMAIIM/FREEMAIN 
VSAM 
ESR (type 2) 
PURGEDQ 
TPIO 
EVENTS 
[no name] 
RACHECK 
RACINIT 
RACF manager 
RACDEF 





•5 

.7 


4.3 

5.9 

2.7 

2.1 
10.5 

.9 
20.4 
3.1 


5.5 
4.9 




.1 




80 

391 

1102 

280 

460 

78 
108 

54 
181 
911 

582 
175 

1662 


216 
1057 
2978 

757 

1243 

211 
292 

146 

489 

2462 

1573 
473 

4492 



Figure IL9 Example of Average TCB Times for SVCs on a 168-3 6 Meg TSO and Batch System (Part 4 of 4) 
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Identifying heavy processor users 

To solve processor (or any other resource) constraints, it is often necessary to 
address the dominant address spaces. Figure 11.10 summarizes the steps described 
in this bulletin for focusing on heavy processor users. 

SMF provides TCB and SRB time for jobs and logons in the type 5 and 35 
records, respectively. Focus on those jobs with the highest TCB time. On a lower 
level, you can identify the programs and commands that use high TCB time. SMF 
provides TCB and SRB time by program in the type 4 (step termination) record. 
Similar information on a command basis is not automatically provided. However, 
if PCF (Programming Control Facility, FDP # 5798-BBJ) is included in the system, 
it can be used to selectively trace commands and to obtain an SMF record for 
commands similar to the type 34 record for TS-step termination. 

TCB time consists essentially of SVC time and problem program time. You can 
identify the SVCs called by a particular job /logon using GTF or the trace table. 
GTFSVC (lUP # 5796-PGE, a GTF reduction program) provides SVC information 
on a "job" basis, where job refers to the jobname field in the GTF trace record 
and therefore includes batch jobs, TSO user sessions, the master scheduler, JES2, 
TCAM, etc. The job detail report further divides SVC usage for a particular job by 
TCB address, module name, and PSW address. Note, however, that no known 
IBM-provided tool reports SVC information on a program or command basis unless 
the program or command equates to a module name(s). 

Using GTFSVC, you can focus on jobs with high percentages of SVC activity 
and then identify the particular SVCs used by those jobs. To judge the usage of the 
specific SVCs requires knowledge of SVC usage you would expect to see, of SVCs 
whose usage might be decreased without an adverse effect on function, and of SVCs 
that are costly in processor time. 

Determining relative processor cost of SVCs is not trivial; a method to do so, 
based on determining the average TCB seconds spent executing each SVC, is 
described in bulletin 2.3.0. If you have determined average TCB seconds for 
each SVC, you can compute approximate SVC time for a particular job via the 
following formula: 

SVC TCB time for job A « 

255 

n=o (SVC n frequency by job A x avg. TCB seconds for SVC n) 
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STEP 


Tool or cross-reference 


1 . Identify users with relatively high TCB time. 


SMF records 5 and 35 


2. Identify progranns or commands with relatively 
high TCB time. (Note, however, that no known 
IBM-provided tool provides SVC information 
on a program or command basis unless the 
program or command equates to a module 
name(s).) 


programs: SMF record 4 
commands: PCF (FDP #5798-BBJ) 


3. Identify jobs with high SVC activity. 


GTF or trace table analysis; 
GTFSVC (lUP #5796-PGE) 


4. Focus on SVC part of TCB time for jobs 
identified via steps 1 and 3. 




5. Determine frequency of SVCs in jobs focused 
on. Judge usage of SVCs, based on: 

• expected or usual usage of SVCs by the 
work in question. 

• SVCs that can be changed or whose 
frequency can be reduced without an adverse 
effect on function. 

• relative processor cost of SVCs 


topic "SVCs" in Part III 

GTF or trace table reduction — see 
bulletin 2.3.0 


6. If relative processor cost of SVCs is computed : 

• determine SVC TCB % for jobs being 
investigated; subtract from total TCB time 
for job to obtain approximate problem 
program time. 

• if problem program time is unusually high 
for type of application, examine program 
design. 

• compute TCB time for particular SVCs used 
by the job and focus on those SVCs that 
account for significant proportion of total 
job TCB time. 


GTF or trace table reduction — see 
bulletin 2.3.0 



Figure 11.10 Steps In Focusing On Heavy Processor Users 
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Identifying heavy processor users (continued) 

Because TCB time is essentially SVC time and problem program time, you can 
then determine problem program time for a particular job by subtracting that 
job's SVC time from total TCB time: 

TCB time for job A - SVC TCB time for job A « problem program time for job A 

If problem program time seems unusually high for the type of application (for 
example, for many IMS applications, SVC time is usually greater than problem 
program time), examine the program or command for design problems, such as 
loops, that could be causing excessive use of the processor. 

In examining the specific SVCs used by the job, you can determine the 
percentage of a job's TCB time used for each SVC in that job : 

% of job's TCB time for SVC Z « 

number of SVC Zs in job x avg. TCB seconds for SVC Z x 1 00 
job's TCB seconds 

By doing this, you can identify those SVCs for which changes or decreases can have 
the most significant impact. 

The topic "SVCs" in Part III contains suggestions on reducing the frequency or 
processor usage of some SVCs. 
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The investigation of I/O resources is a continuous process. However, when system- 
oriented performance problems are occurring, I/O bottlenecks are the likely culprit 
— and one of the first areas to focus on — when processor wait time is high (for 
example, 10-15% of elapsed time, as reported on the RMF or MF/1 CPU activity 
report). The means of confirming that processor wait time is due to I/O delays is 
essentially the means of identifying solutions to I/O delays. 

I/O delays are also one of the causes of user-oriented performance problems. 
Bulletin 5.4.0 in Chapter 11.5, "User-oriented Performance Problem Analysis," 
describes how to determine if specific work is suffering due to I/O delays. Possible 
I/O bottlenecks and ways to determine which are occurring, however, are 
documented here, in this chapter. The major difference in investigating I/O delays 
that are impacting user-oriented objectives (as opposed to system-oriented 
objectives) is that you have already focused on specific volumes experiencing I/O 
delays. Finding a solution that will not cause excessive delays on other volumes or 
adversely affect other work, however, usually involves the investigation of the 
entire I/O resource configuration. 

The goal of investigating I/O resources is to minimize delays in satisfying I/O 
requests. There are six steps required to satisfy an I/O request and each 
of these steps has performance implications, as illustrated in figure 11.11 . 



STEP 


Impacted by: 


Solutions 


1. WAIT for device to become 
ready 


volume mounting, forms 
loading on printers, etc. 


operational procedures 


[issue EXCP] 






2. WAIT for I/O path 
[issue SIO] 


overall utilization of 
channel, control unit, and 
device 


data set placement across 
devices; eliminate bottle- 
neck in path; reduce 
number of EXCPs/SIOs 


3. SEEK 


arm movement 


data set placement on 
volume 


4. ROTATIONAL DELAY 


speed of device 


choice of device 


5. RECONNECT 
(RPS devices) 


missed reconnects due to 
channel/ control unit busy 


balance use of channel/ 
control unit; reduce 
number of EXCPs/SIOs 


6. DATA TRANSFER 


inefficient btocksizes 


examine bio cksizes 



Figure II. 1 1 Steps in Satisfying I/O Requests and their Performance Factors 
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Step 1 delays occur at OPEN and allocation time, before the EXCP is issued. As a 
result, delays at this step are independent of the other steps. Steps 2 through 6 are 
related. Steps 3,4, and 6 pertain specifically to the individual request — the device 
time required to service the request. Steps 2 and 5 pertain more generally to the 
overall utilization of the resources needed to satisfy the request. However, high 
device service time (steps 3, 4, and 6) for individual requests can cause delay at 
steps 2 and 5 for other requests to the same device: utilization involves both the 
amount of resource activity (the number of requests) and the time required to 
service the requests. Because of the relationship of the steps, I/O delays usually 
cannot be attributed to a single step, a single cause; or solved by a single solution. 

Operational bottlenecks — delays occurring at step 1 — and possible solutions 
to them are described in the topic "Operational Bottlenecks" in Chapter 1.3, 
"Pre-initialization MVS Performance Factors." 

Of the remaining steps, steps 2, 3, and 5 have the greatest potential for delay 
and, therefore, the greatest potential for performance improvement. (Rotational 
delay is almost inevitable and data transfer time is often a small portion of the 
total time required to service a request.) As a result, the most significant means 
of improving I/O performance are data set placement (on a volume and across 
devices), reducing the number of EXCPs/SIOs, and reconfiguration of the resources. 
This chapter does not try to describe how to configure I/O resources, but rather 
how to identify critical I/O paths and possible bottlenecks in the present 
configuration — which might indicate the need for reconfiguration or additional 
hardware resources, if changes in data set placement are ineffective. The following 
topics are described in Bulletins 3.1,0, 3.2.0, and 3.3.0, respectively: 

• identifying critical I/O paths 

• focusing on specific I/O bottlenecks 

• reducing bottlenecks in the I/O path. 

Bulletin 3.4.0 describes reducing the number of EXCPs/SIOs. 

Figure 11.12 summarizes the process of investigating I/O resources, as described 
in this chapter. 
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TASK 


Cross-reference 


1 . Investigate possibility of operational bottlenecks. 


"Operational Bottlenecks" in 
Chapter 1.3 


2. Focus on possible critical I/O paths: 

A. Identify potential problem channels: 

• "percent channel busy and CPU wait" 
measurement 

• "percent channel busy" measurement 

B. For potential problem channels, determine on 
which devices requests are delayed: 

• "queue length" measurement 

• average time to service a request 

At this point, possible critical I/O paths are 
identified, but specific bottlenecks are not. 


Bulletin 3.1.0 


3. Check for specific bottlenecks: 

A. Device bottlenecks: 

• RESERVE interference from other processor 
(shared DASD) 

• high SEEK time 

B. Channel bottlenecks: 

• high channel time per access 

• contention caused by reconnects from 
RPS devices 

C. Control unit bottlenecks: 

• formatted writes and activity on alternate 
channels 


Bulletin 3.2.0 


4. Balance contention by changing data set placement 
and reconfiguring resources 


Bulletin 3.3.0 


5. To reduce contention (as opposed to balance 
contention), reduce number of EXCPs/SIOs 
and/or use workload management to separate 
the work responsible for contention. 


Bulletin 3.4.0 



Figure II. 12 Summary of Process for Investigating I/O Resources 
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Identifying critical I/O paths 

RMF and MF/1 provide data on the utiUzation of channels and devices. The 
following measurements (from the channel and device activity reports) can be used 
to identify potentially critical I/O paths: 

• "percent channel busy and CPU wait " can indicate where delays are occurring 
that might be contributing to processor wait time. 

• "percent channel busy " gives the utilization of channels and can indicate 
relatively over- or under-utilized channels. 

• "queue length " for devices might indicate high wait times for satisfying requests 
on specific devices. Using queue length and other data on the RMF or MF/1 
device activity report, you can compute average time to service a request for 
requests to the devices on the potentially critical channels. 

Each of these measurements is described in more detail at the end of this bulletin. 
Note that the channel activity report reports only on physical channels — primary 
channel activity is not distinguished from alternate channel activity. 

Although these measurements can be used to focus on possible critical I/O 
paths, they do not indicate possible specific bottlenecks — that is, the specific 
resource in the path (channel, control unit, or device) that might be causing the 
delay. Measurements that can indicate specific bottlenecks are described in 
Bulletin 3.2.0. 

In judging the criticalness of I/O paths, you must consider the data being 
accessed by the path. Usually the goal of I/O analysis is not to absolutely balance 
I/O resource utilization — for example, all channels experiencing the same utiliza- 
tion. The criticalness of the data on the I/O path is an important factor in judging 
these measurements. 

Details on each of the measurements noted above follow. 

"Percent Channel Busy & CPU Wait" Measurement 

The "percent channel busy and CPU wait" measurement is the percentage of 
elapsed time (in the report interval) during which the channel activity did not over- 
lap processor activity. If the values for this measurement are significant and there is 
sufficient work in the system, you can conclude that I/O delays are causing 
processor wait time. However, if the values are not significant, you cannot 
conclude the opposite — that I/O delays are not causing processor wait time. 
Channels usually disconnect, except during data transfer (an exception being 
charmel programs that do not issue SET SECTOR for RPS devices); therefore, 
although a channel is not busy, I/O delays can be occurring at the control unit or 
device level. 
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Identifying critical I/O paths (continued) 

Some performance analysts combine this measurement with "percent CPU 
wait" (from the RMF or MF/1 CPU activity report) to obtain the percentage of 
processor wait time during which a channel was busy: 

percent channel busy and CPU wait mn 
percent CPU wait 

Results of this computation might be clearer than the "percent channel busy and 
CPU wait" measurement by itself. For example, processor wait time is 10% of 
elapsed time ; channel busy for channel A is 20% of elapsed time ; "percent channel 
busy and CPU wait" for channel A is 8. Using the preceding formula, the 
percentage of processor wait time during which channel A was busy is: 

-?- X 100 = 80% 
10 

In this example, because of the high percentage, it is very likely that channel A is 
contributing to processor wait time. 

"Percent Channel Busy " Measurement 

Judgment that a channel-busy measurement is high or low must be made in light 
of the type of device (DASD or tape) and the type of requests to that device (that 
is, the data sets on the device — for batch or for interactive work). For example, 
many performance analysts consider 33% to be the upper boundary for block 
multiplexors and 66%, for selector channels. However, in a response-oriented 
system (such as CICS), 20% might be "high" channel utilization on a block 
multiplexor; but, if throughput is the most important criteria, acceptable channel 
utilization could be as high as 50%. Likewise for selector channels: for tape 
devices in a batch, throughput-oriented environment, you might want to achieve 
channel utilizations much higher than 66%. 

Queue Length 

High queue lengths for a device (available on the RMF or MF/1 device activity 
report) might indicate high wait time for satisfying requests on that device. The 
judgment of whether a queue length is "high" depends on its relationship to the 
queue lengths on other devices and to the data accessed on that device (for example, 
by batch or interactive work). As a starting point, many performance analysts 
consider a queue length to be high if it equals or exceeds 10% of device busy 
expressed as a fraction. For example, on a device with 40% busy, queue lengths 
of .04 or higher might indicate high wait time for satisfying requests on that 
device. 
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Identifying critical I/O paths (continued) 

Note, however, that the queue length measurement itself does not indicate how 
long requests must wait. The average time to service a request (described next) is 
more likely a more reasonable indicator of I/O delays since it actually indicates 
how long requests must wait. 

Average Time to Service a Request 

The average time to service a request on a device is a reasonable indication of 
devices that are experiencing I/O delays. This time (from the STARTIO macro to 
I/0-complete) can be computed via the following formula: 

(% device busy — 100) + queue length report internal _ average time to 

device activity count (in seconds) service a request 

where each factor is available on the RMF or MF/1 device activity report. {Note: 
If the activity count is low, the result might not be meaningful.) This formula is a 
reduced version of the following: 

(% device busy 4-100) X report interval in seconds 
device activity count 

, queue length X report interval in seconds 

+ _2 r L — = average service time 

device activity count 

where the first factor is the time to process the request and the second is wait time, 
average service time being the sum of the two. Using this version of the formula, 
you can break average service time into processing time and wait time. 

Judgment of the computed service time is subjective and should be based on 
comparison to service times on other devices and the speed of the device. Ideally, 
the time to satisfy a request should be determined by data transfer time (which is a 
factor of the speed of the device), as opposed to contention or seek time. Although 
contention and seek time are essentially inevitable, time to satisfy a request should 
still be fairly consistent for like devices accessed by similar applications and relative 
to the data transfer rate for that device, as compared to other device types. 
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Focusing on specific I/O bottlenedcs 

The measurements described in Bulletin 3.1.0 can be used to focus on probable 
critical I/O paths; however, they do not indicate the specific bottleneck in the path 
- channel, control unit, or device. The following topics describe some specific 
possible bottlenecks at each point in the I/O path and measurements or tools that 
can be used to investigate the possible bottleneck. 

Device Bottlenecks 

Bottlenecks at the device level can be caused by: 

• RESERVE interference from another processor if the device is shared. This is 
often indicated by high device utilization accompanied by a low device activity 
count (reported on the RMF or MF/1 device activity report). 

• excessive seek time. This might also be indicated by high device utilization 
accompanied by low device activity. GTF reduction programs (for example, 
SEEKANAL, lUP # 5796-PJC) can provide information on seek patterns. 
Examine VTOC and AMS listings for the packs in question. See the topic 
"Seek time, reducing" in Part III. 

Note that high device utiUzation by itself is not necessarily undesirable — the 
significant measurement is average time to service requests on the device (which, 
however, does not indicate the specific source of bottlenecks; Bulletin 3.1.0 
describes this measurement). 

Channel Bottlenecks 

Channel bottlenecks can be caused by the following: 

• high channel time per access. Channel time per access is usually only a fraction 
of device time per access. High channel access time — for example, greater than 
one-half of the average device access time experienced by the devices on that 
channel — can indicate a channel bottleneck. You can use the following 
formulas to compute channel and device time per access: 

channel time per access = -^ X report interval in seconds 

channel activity count 

. . ,. (%devicebusy -f 100) ., ^.^ ,. . 

device time per access = X report interval in seconds 

device activity count 
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Focusing on specific I/O bottlenecks (continued) 

where each factor is provided by the RMF or MF/1 channel or device activity 
reports. High channel access time can be caused by the failure of channel 
programs to use the SET SECTOR instruction for RPS devices - reviev/ 
installation-written channel programs. 

• contention caused by reconnects from RPS devices. This can be indicated by an 
unusual number of SIOs resulting in condition code 2, which indicates the 
channel was busy. (GTF reduction programs — for example, SEEKANAL, 

lUP # 5796-PJC, and GTFIOCUR, lUP # 5796-PGD, can be used to report SIC 
condition codes.) Before issuing an SIC, lOS first confirms that a channel is 
available. If the SIC returns condition code 2, it means that, between the time 
lOS found the channel available and issued the SIO, the channel became busy 
due to reconnect from an RPS device. If there is no alternate channel, an 
unusually high number of condition code 2s indicates the channel is over- 
utilized. If there is an alternate channel, this condition can reflect uneven 
utilization of the primary and alternate channels. lOS always tries first to use 
the primary channel path (the one specified first at sysgen). This usually results 
in the primary channel path being more highly utilized than the alternate path, 
which can impact requests to RPS devices. Condition code 2 reflects this 
contention for requests going out to the device. The topic "Channel balancing" 
in Part III includes suggestions on balancing activity between primary and 
alternate channels. 

No simple measurement is available that reflects the inability of a request to 
reconnect due to high channel utilization or control unit contention. 

Control Unit Bottlenecks 

Usually the control unit is busy at the same time the channel is busy. There are 
two basic exceptions: 

• During format write operations when the CCW is the last CCW on the string, the 
channel is free while the control unit is busy. 

• If a control unit is attached to more than one channel, the control unit can be 
busy with activity on the other channel. 

Both of these conditions result in condition code 1 being returned for the SIO. 
(lOS checks only that the device and channel are available before issuing an SIO; 
it assumes the control unit is available if the channel is available.) Because of this, 
some performance analysts consider the control unit to be the probable bottleneck 
in an I/O path if more than 20% of the SIOs to a device result in condition code Is. 
However, condition code 1 means that the CSW was stored and does not reflect 
only control unit busy conditions. This indicator of possible control unit 
contention must be considered in light of the following: 
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Focusing on specific I/O bottlenecks (continued) 

• It should not be applied to devices shared between systems. CC=1 will be 
returned if the device is being used by the other system (busy bit turned on in 
CSW); this has as great a probability of occurring as the control unit being busy 
(busy and status modifier bits turned on in the CSW). 

• CC=1 is returned for immediate operations, such as tape rewinds and NOOPs. 
For tape devices, you might want to consider a higher proportion of SIOs 
resulting in condition code Is as an indication of a probable control unit 
bottleneck. 

Although other conditions can result in CC=1 (for example, channel errors), the 
likelihood of their occurrence is not usually significant. 
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Reducing bottlenecks in I/O paths 

The process of reducing bottlenecks in I/O paths is essentially experimental, the 
"trial-and-error" method — for example, moving data sets to different packs, 
moving packs to different devices — and remeasuring. The measurements described 
in Bulletins 3.1 .0 and 3.2.0 can be used to identify possible critical I/O paths and 
specific bottlenecks and therefore guide the changes in data set placement or 
reconfiguration. Equally important as the clues to possible existing bottlenecks, 
hovi'ever, is a thorough grasp of the data being accessed: characteristics of the 
data sets, including their frequency of use ; the time of their use ; the users of the 
data sets and the stringency of the performance objectives for those users. Knowing 
the EXCP rates of specific users can help guide decisions for workload balancing to 
reduce I/O contention. (Bulletin 3,4.0 describes ways to compute EXCP rates.) 

When using data set placement to reduce I/O bottlenecks, beware of the myth of 
low-utilized data sets. All data sets are highly utiUzed when they are used; some 
are highly utilized more frequently than others. The decision to place infrequently 
used data Sets on volumes with frequently used data sets must be based on how 
critical the frequently used data sets are and when the infrequently used data sets 
are used. For example, if the time required to resolve page faults is critical to the 
performance of your system, the presence of an infrequently used data set on the 
paging pack might be disastrous, depending on when it is used: during peak periods 
or during off-shifts when the workload is light and performance objectives are not 
stringent. 
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Reducing EXCPs/SIOs 

Reducing the number of EXCPs/SIOs has two potential benefits: reduced 
contention for I/O resources due to fewer requests; and reduced processor time. 

You can use the following formula to focus on work with high EXCP/SIO rates 
and to determine the result of actions taken to reduce EXCP/SIO rates: 

EXCP/SIO rate = "° '"''"'' 



I/O coefficient x RIVIF interval in seconds 

where I/O service and the I/O coefficient are reported on the RMF workload 
activity report. This formula can be applied on a system level or to any of the 
discrete breakdowns of work on the workload activity report (for example, 
performance groups, domains). The EXCP counts reported by RMF are those 
that are counted in SMF; appendix B in 0S/VS2 SPL: System Management 
Facilities summarizes the I/O activity included in and excluded from the EXCP- 
count fields of SMF records. 

To reduce EXCPs to the paging data sets, reduce the amount of demand paging 
- see bulletin 5.2.0 in chapter II.5. 

For TSO and batch work, SMF provides EXCP counts on a job-step and TSO- 
user basis in records 4 and 34, respectively. Ways to reduce EXCP rates for TSO 
and batch work include the following: 

• Consider using VIO for TSO and batch temporary data sets. The Initialization 
and Tuning Guide contains suggestions on the use of VIO data sets; additional 
hints will be added to Part III of this book in a future update. 

• Increase blocksizes — see the topic "Blocksizes and Buffers" in Part III. 

For IMS, examine the ratio of data-base SIOs to DB calls or data-base SIOs per 
transaction. To obtain the number of SIOs to the data base, add the device activity 
counts on the RMF or MF/1 device activity report for the devices containing the 
data base (assuming the data base is on dedicated devices). Total DB calls or total 
number of transactions can be obtained from the appUcation accounting reports 
produced by the IMS/VS Statistical Analysis Utility Programs. (Reduce a subset 
of the IMS/VS log tape over the same time interval as that for which RMF data is 
collected.) Since a DB call might access data that is still in the IMS/VS data base 
buffers, there can be more DB calls than SIOs. 
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Reducing EXCPs/SIO (continued) 

The IMS/VS DC Monitor reports on data base buffer pool activity and on 
message queue pool activity, from which the total number of I/Os done to 
satisfy pool requests can be computed. The DC Monitor can also be used to 
compute the total number of transactions and total number of DL/I (DB and DC) 
calls executed in that interval. Then, the number of message queue pool I/Os per 
transaction (or per DL/I call) and the number of data base I/Os per transaction 
(or per DL/I call) can be computed. The DC Monitor also reports on IWAITs per 
DL/I call. (Note that, for IS AM/OS AM data bases, IWAITs translate to SIOs; 
for VSAM, IWAIT indicates a call to VSAM, which does not necessarily mean that 
I/O was performed.) 

Ways to reduce SIOs to the data base include the following: 

• Experiment with larger buffers, but beware of possible performance trade-offs 
when doing so. Although this might result in reduced SIOs, it can also result 
in increased processor time searching the buffers and additional buffer paging, 
both of which can impact IMS performance. 

• If needed, perform an IMS/VS data base reorganization. See the IMS/VS 
Utilities Reference Manual. 

• For IS AM/OS AM data bases, consider the following: 

— Use the ISAM "index-in-core" option by specifying DCB=OPTCD=R on the 
DD statements defining ISAM data sets. This loads the highest level index of 
the ISAM data set into IMS/VS control region storage. 

— Eliminate ISAM/OSAM data base compaction by making all data base block- 
sizes equal or by specifying the BFPLBFSZ operand for the DFSVSAMP 
data set equal to the largest ISAM/OSAM data base blocksize plus 48 bytes 
for the buffer header. 

• For VSAM data bases, consider the following: 

— Allocate VSAM data base buffers within a subpool to be equal to the control 
interval size of the data base. For example, if your data base has control 
interval sizes of IK, 2K, and 4K, allocate three subpools with IK, 2K, and 4K 
buffers, respectively. This reduces wasted buffer pool space: if you have IK 
control intervals (for example), and 8K buffers, the system wiU use only IK 
of each buffer. VSAM data base blocks will be allocated to the buffer with 
the best fit. Specify the number of buffers in each subpool according to the 
activity of that size data base record: if 2K records are accessed much more 
frequently than 4K records, define more buffers in the 2K-buffer-size 
subpool. 

— For data sets in a VSAM data base, specify a percentage of free space and a 
percentage of empty control intervals per control area when defining the 
VSAM data sets. This will reduce the time required to insert segments as 
well as centralize related segments. 
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Paging is the mechanism for managing real storage and paging rates are usually 
examined to determine if a real storage problem exists."*^ The total paging rate by 
itself will not necessarily indicate if real storage is the critical resource contributing 
to a performance problem. Paging rates vary widely and it is possible to have a 
"high" rate (as compared to other installations) and good performance. The total 
paging rate can be divided into demand paging (non-swap non-VIO page-ins and 
page-outs), swap paging, and VIO paging rates. (See figure 11.13, a sample RMF 
paging report included for your reference.) Each of these types of paging is 
independent of the others in that solutions for decreasing the rate of one will not 
usually decrease the rate of another. When examining each of these rates, many 
performance analysts compute the rate as page-ins and page-outs, excluding 
reclaims. Reclaims are controlled by the page-replacement algorithm and 
investigating reclaim rates on a system level usually does not lead to solutions of 
performance problems. (Reclaim rates on an address space level, however, can 
indicate an appUcation's mis-use of its own address space.) 

As with total paging rates, demand, swap, and VIO paging rates vary widely 
and cannot necessarily be judged by themselves. Paging rates must be examined in 
terms of their impact on the system as a whole; paging can cause performance 
problems in the following ways: 

• Paging I/O is interfering with other I/O in the system. 

• Paging is costing in terms of non-productive processor time. 

• Paging is causing processor wait time — work ready to be dispatched must wait 
for page faults to be resolved. 

Investigation of real storage should be triggered by recognition of one of these 
situations, each of which is described in more detail below. 

Paging I/O Interfering with Other I/O 

This situation is indicated by an inability to balance channel, control unit, and 
device contention due to paging activity. The approach to remedy this situation 
is to reduce activity to the page data sets - and, therefore, decrease the interference 
of paging I/O with other I/O. To determine the area of most potential benefit - 
demand, swap, or VIO paging - use the RMF paging activity report to compute 
the percentage of each type of paging. For example, a total paging rate of 45.32 
(from the sample report in figure 11.13) breaks down as follows: 

• demand paging - 3.99 - 8.8% of total 

• swap paging - 40.57 - 89.5% of total 

• VIO-.76- 1.7% of total 



Some performance analysts used to examine the number of frames on the available frame 
queue as an indication of real storage problems. With SU7 (Supervisor Performance #2), 
the SRM keeps the available frame queue close to the AVQOK value and, therefore, it is 
no longer a valid indicator. 
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Figuie 11.13 Sample Paging Rates from RMF Paging Activity Report 



In this case, decreasing swap pages has the greatest potential benefit; VIO paging is 
negligible. Each of these types of paging is described in more detail in the bulletins 
at the end of this chapter. 



Paging Costing Non-productive Processor Time 

This situation can be indicated by high processor utilization accompanied by a 
seemingly high paging rate. A more precise way to recognize this situation is to 
use the formulas for breaking down non-TCB processor time (see bulletin 2.2.0 
in the chapter II.2, "Investigating the Use of Processor Time"), which give the 
approximate percentages of processor time used for swapping and for non-swap 
(demand and VIO) paging. If non-swap paging processor time seems high, you 
can focus specifically on VIO or demand paging by looking at their respective 
paging rates on the RMF paging activity report. 

The approach to remedy this situation is to decrease the paging rate of the 
type of paging that is using most processor time : swapping, demand, or VIO, 
as determined by use of the formulas and comparison of demand and VIO 
rates. The bulletins at the end of this chapter describe reducing each of the types 
of paging. 



Paging Causing Wait Time 

Work ready to be dispatched must wait for page faults to be resolved. This 
situation is more difficult to detect than the others. A symptom of this situation is 
high demand paging rates. However, high demand paging can also be the result of 
wait time rather than the cause of wait time : if an address space is waiting, pages 
can be stolen because they are unused and have to be paged back in when the work 
is dispatched again. Although a high demand paging rate, therefore, does not 
necessarily mean paging is causing wait time, it can be used to trigger further 
investigation of real storage. See Figure 11.14 for approximate target values for 
demand paging. To remedy this situation, reduce demand paging and/or reduce the 
time required to resolve a page fault — see bulletins 4.2.0 and 4.4.0, respectively. 
For IMS systems, see also the topic "Demand Paging in IMS" in Part III. 
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Note: Paging rates are high only if they are adversely affecting performance, as described 
in the text. Because it is difficult to determine if paging is causing wait time, these target 
values are included as a starting point: if demand paging exceedsthe target values, you 
might want to further investigate demand paging. However, higher values might be 
appropriate for your installation. The values in this figure should be used only as a rough 
guideline, not as a goal. 


TSO dedicated 
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158 UP - 10 
158 MP - 17 
168 UP - 25 
168 MP - 43 


IMS* dedicated 
IMS* non-dedicated 


158 UP - 5 
158 MP - 9 
168 UP - 15 
168 MP - 26 


*Pre-load in IMS can result in higher paging rates. Depending on the number of applica- 
tions pre-loaded, you might expect to see higher demand paging numbers. Also, IMS 
demand paging target values should be considered in light of your particular installation 
and the IMS response time objective. Since a page fault in the IMS control region 
affects all IMS terminals and a page fault in a message processing region affects all the 
terminals serviced by that region, you have to consider the effect of the delay tp resolve 
a page fault. For example, on a 168, where more processing regions probably exist, 
you might be ^le to tolerate paging more easily than on a 158, with fewer message 
processing regions defined. 



Figure II. 14 target Values for Demand Paging 
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Swapping 

The RMF paging activity report Usts the number of swaps according to the type of 
swap. Solutions to reduce swapping depend on the type of swap being 
experienced : 

• input terminal wait swaps are "productive" swaps and, therefore, not usually 
of concern. They reflect the transaction rate and the only way to reduce their 
rate is to reduce the transaction rate - which is not usually desirable. 

• output terminal wait swaps should be close to zero. They can occur for two 
reasons: 

- the TIOC does not have sufficient output buffers, because TSO commands or 
CLISTs are producing unusual amounts of output or because sufficient buffers 
were not allocated initially. The SRM treats this condition as the end of a 
transaction and automatically swaps out the address space until the output 
buffers are emptied. If swaps occur consistently for this reason, evaluate the 
OWAITHI value in the IKJPRMxx PARMLIB member. 

- application programs are issuing TPUT SVCs with the HOLD keyword. If 
this is true, re-examine the necessity of coding HOLD. 

• requested swaps occur automatically when V=R or non-swappable jobs or steps 
are started. The only way to reduce these is to reduce the amount of non- 
swappable work, or work running V=R. 

• auxiliary and real pageable storage shortage swaps should not normally occur 
and, if they do, can signify a potentially serious problem. For auxiUary storage 
shortage swaps, first check the job swapped out (it is identified at the system 
console) for a VIO loop — VIO loops can cause excessive auxiliary page slots to 
be allocated. Otherwise, these types of swap indicate the need for additional 
hardware resources (auxiliary or real storage, depending on the type of swap), 
unless one of the following is possible : 

— the workload can be reduced and/or critical address spaces (those that 
acquire auxiliary storage page slots or fixed frames at the greatest rate) can 
be moved to non-critical shifts. 

— more auxiliary storage can be assigned to the page data sets or fixed require- 
ments in real storage can be reduced to make more pageable storage available. 

• long wait swaps usually occur as a result of the STIMER macro or enqueue 
contentiQn — in the first case, reduce use of STIMER; in the second, identify 
and reduce or eliminate the source of contention. Long wait swaps can also 
occur during MOUNT processing (waiting for a DASD volume to be mounted) 
and during allocation recovery (waiting for a reply to a WTOR). 
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Swapping (continued) i 

• unilateral and exchange-on-recommendation-value swaps are based on the IPS 
specification. Reducing these involves modifications to the IPS specification — 
see the Initialization and Tuning Guide. 

• enqueue exchange swaps are reduced by identifying and reducing the source of 
contention. 

• detected wait swaps occur when an address space in storage has been non- 
dispatchable for a fixed period of time. Solutions depend on why the address 
space is nondispatchable; for example: 

- an initiator runs out of work. Re-examine the number of initiators and the 
classes assigned to each. 

- the address space is enqueued on a resource in use by another address space. 
Identify and reduce or eliminate the source of contention. 
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Demand paging 

High demand (non-VIO non-swap) paging indicates the possible over-commitment 
of real storage — that is, too many users competing for real storage. If sufficient 
frames are not available on the available frame queue to assign to users as they 
require more storage, pages are stolen (according to the unreferenced interval 
count) to free required frames. If the pages stolen are referenced again, they must 
be reclaimed (if they have not been paged out) or paged-in (demand paging). 



Steps to Reduce Demand Paging 

There are several steps that can be taken to reduce demand paging: 

1 . Increase the size of pageable storage by reducing the number of modules, data 
areas, and control blocks that are fixed. 

2. In the pageable areas (PLPA and private), examine module packing and CSECT 
ordering to try to achieve efficient page reference patterns: in terms of density 
of reference (so that fewer pages are active at one time) and frequency of 
reference (so that pages that are stolen due to a high unreferenced interval count 
are not re-referenced). 

3. Identify users with exceptionally large working sets and investigate workload 
changes to avoid the concurrent execution of these users. 

4. Reduce the multiprogramming level. 

Each of these steps is described in the following topics. To visualize how storage is 
being used and to help focus on specific areas, it is useful to draw a storage map of 
the system. The RMF paging activity report provides the number of frames 
assigned to the following areas: 

• SQA (always fixed) 

• LPA/CSA - fixed 

• LPA/CSA - pageable 

• private area — fiixed 

• private area — pageable 

• available (unused) frames 

• nucleus. 

SIR can be used to break down the private area by user. 
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Demand paging (continued) 

Reducing Fixed Storage Requirements 

The feasibility of and approach to reducing fixed storage requirements depends on 
the area that is fixed: 

• The nucleus can be reduced by generating fewer options - for example, do not 
specify ACR (alternate CPU recovery) if the system is not MP; remove excess 
unused UCB definitions. 

• The SQA is difficuh to reduce. Identifying the users responsible for fixed 
frames in SQA requires dumping the SQA and trying to trace who was 
responsible for the various entries in the system control blocks. It is not always 
feasible to reduce SQA, and because the effort to try is significant, SQA is not 
a prime candidate for reduction. 

• The amount of CSA fixed depends on the environment. In batch or TSO 
environments, almost none of CSA is fixed. In IMS environments, the amount 
fixed depends on the IMS fix options chosen by your installation and the size 
of the pools. Because IMS pools are fixed to improve I/O performance and 
response time, reducing the amount fixed in order to reduce demand paging 
involves trade-offs: you must assess the potential impact on I/O performance 
and response time of reducing fixed CSA. 

• To reduce fixed LPA, examine the choice of modules in the fix list (lEAFIXxx 
member of PARMLIB) to determine if reentrant modules might better be 
placed in pageable LPA. If a module is frequently used, it will likely remain in 
storage, although it is placed in PUPA. 

• In the private area, every address space will have a minimum set of fixed frames 
— usually five or six — for LSQA. Focus on those address spaces whose fixed 
areas greatly exceed this minimum set. (SIR reports the number of fixed frames 
assigned to each address space.) Solutions to reducing the fixed areas depend on 
the user and what is being fixed. For example, TCAM often causes a large 
number of frames to be fixed for buffers; these can be reduced by reducing the 
number and/or size of the buffers, although possible impact on I/O performance 
must also be considered - see the topic "TCAM" in Part III. 



Page Reference Pattern of Pageable Modules 

Paging can be decreased in PLPA and the private areas by efficient page reference 
patterns. The goal is to localize page references in terms of the number of pages 
required at a time (density of reference) and in terms of time (frequency of 
reference, so that a page that might be stolen after disuse is not referenced again). 
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Demand paging (continued) 

RMF and MF/1 report demand paging in the private area and in the system area 
(PLPA and pageable CSA — the SU7 version of SIR provides a summary breakdown 
of PLPA and CSA paging). Focus on the area that experiences the greater 
proportion of demand paging. SMF records 4 and 34 and SIR can be used to 
break down demand paging in the private area by individual address space; 
SIRTSO provides this information dynamically. Also examine reclaim rates, 
available in SMF records 4 and 34 ; a high page reclaim rate indicates repetition of 
a page reference pattern after periods of disuse. 

No known IBM-provided tool reports specifically on page-reference patterns. 
Having focused on PLPA or individual address spaces via RMF and SMF data, study 
module reference patterns and CSECT ordering within modules to try to identify 
specific solutions. 

Separating Users with Large Working Sets 

Working set refers to the number of fixed and pageable frames assigned to a user in 
the private area. Working set sizes are available from various sources; for example: 

• SIR reports working set sizes. 

• You can use RMF to compute working set size for any user that exists by 
itself in a performance group, via the following formula: 

MSO service x CPU coefficient x 50 



average frames = 



MSO coefficient x CPU service 



The variables are reported on the RMF workload activity report. Note that the 
formula is invalid if the MSO coefficient is zero; if your installation normally 
sets the MSO coefficient to zero, a small MSO coefficient can be used. This 
formula is based on the formula used by SRM to derive main storage service: 

MSO _ iVISO coefficient x CPU service (at coefficient of 1) x average frames 
service 50 

• SMF records 4 and 34 contain data by which you can compute average working 
set size for each job step or time-sharing step: "number of page seconds" in 
miUiseconds (actually 1 ,024-microsecond units) in the relocate section and 
"CPU time under TCBs" in hundredths of a second in the accounting section. 
Adjust each of these to microseconds and divide the number of page micro- 
seconds by CPU microseconds. For example: 

page TCB CPU milliseconds = 6470 
TCB CPU = 1 7 hundredths of a second 

Therefore: 

page TCB CPU microseconds = 6470 x 1024 = 6,625,280 
TCB CPU microseconds - 17 x 10,000 = 170,000 
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Demand paging (continued) 

And: 

• • ^ • 6,625,280 oo n-i .«^ on 

average working set size = -^ = 38.97 « 39 pages 

170,000 

Focus on those users with exceptionally large working sets. If more than one 
batch job has a large working set, try to execute them separately; for example: 
assign the large jobs to a single domain with a maximum MPL of 1 ; or assign 
them to a unique job class and assign that job class to only one initiator. Ways 
to reduce working set size are reducing fixed requirements in private address 
spaces and ensuring efficient page-reference patterns — as described in the previous 
topics. 



Reducing the Multiprogramming Level 

If no batch jobs have exceptionally large working sets, and reducing fixed 
requirements and investigating page reference patterns do not result in sufficiently 
reducing demand paging, lower the multiprogramming level in the system. This can 
be done by lowering the MPL for specific domains, which allows you to control 
what work will have the lower MPL. On a system level, it is also possible to achieve 
a lower multiprogramming level by changing the threshold values for the page fault 
rate and the UIC (unreferenced interval count). However, unlike changing MPLs, 
this cannot be done via an external control (it requires zapping constants; see the 
Initialization and Tuning Guide for details on these constants). For additional 
information on changing the MPLs or zapping constants to achieve lower demand 
paging rates in IMS mixed environments, see the topic "Demand paging in IMS" 
in Part III. 
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VIO paging 

The first step in investigating high VIO paging rates is to review VIO usage to 
determine if it is being used effectively ; see the Initialization and Tuning Guide 
for VIO performance considerations. (Additional hints on VIO usage will be added 
to Part III of this book in a future update.) If VIO paging must be reduced, 
identify the data sets using VIO. Records 4 and 34 in SMF identify the VIO page- 
ins on a user level. Focus on those users with a relatively high number of VIO 
page-ins and identify the VIO data sets being used by these users. Consider using 
normal allocation for the VIO data sets that experience a high VIO paging rate. 
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Reducing the time to resolve page faults 

Several measurements have shown that performance is degraded if a page-in 
requires, on the average, more than approximately seventy milliseconds to resolve. 
The following formula can be used to determine the approximate amount of time 
required to resolve a page-in in your system: 

RCVASMQA 

== average time per page- in 



total system paging rate 



where RCVASMQA (ASM queue length average) is provided by the RMF trace and 
total system paging rate is reported on the RMF paging activity report. (This 
formula can also be used to compute the average time to resolve a page-out.) The 
time computed via this formula is from when ASM receives a page request until 
I/O is complete and the requestor is posted. Note, however, that the requestor 
will not necessarily be dispatched immediately after being posted ; the requestor's 
wait time, therefore, can be longer than the time to resolve the page-in. 

If page-ins are taking excessive time to resolve, examine the number and 
placement of page, PLPA, and common data sets — see the Initialization and 
Tuning Guide for suggestions on the placement of paging data sets. 
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User-oriented objectives include response time for interactive transactions and 
turnaround time for batch. Use the analysis described in this chapter to address 
the following problems : 

• Response time is inadequate for a particular class of transactions (for example, 
TSO trivial transactions). 

• Turnaround time is inadequate for a particular class of jobs. 

• Turnaround time is inadequate for a specific application. 

The analysis is aimed at answering, "Why is the work in question waiting (that is, 
not receiving the service it requires to meets its objective) ?" 

Identifying Why Work is Being Delayed 

The major difficulty in solving user-oriented performance problems is determining 
why the work in question is not receiving the service it requires to meet its 
performance objective. This requires measurements on an address-space level and 
often requires the use of GTF data or the trace table to trace and time events 
attributable to the work in question. 

There are several major causes of delay: 

• waiting to start or excessive input queue time. 

• while in storage, waiting for the processor. 

• excessive demand paging. 

• waiting for I/O to complete (including delays due to operational bottlenecks). 

• enqueue contention for TSO or batch work or, for IMS work, data base 
contention. 

• excessive swapping (applicable to TSO and batch work only, since the IMS 
control region must be nonswappable and it is recommended that the IMS 
message processing regions be made nonswappable; however, excessive swapping 
of TSO or batch work can affect IMS response time, due to system interference). 

• contention for locks and IMS latches. 

With the exception of locks and latches, investigation of each of these is described 
in the bulletins at the end of this chapter. (Information on investigating lock/latch 
contention will be included in a future update.) 
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There is no firm order in which to investigate the possible causes — if a specific 
resource is critical in your installation, investigate causes related to it first. For 
example, if storage is critical, start by checking swapping, excessive paging, and 
waiting to start. However, beware that you cannot always focus on one cause until 
others have been eliminated. For example, if the work is in storage and is not 
obtaining sufficient access to the processor, you cannot assume the work is being 
denied access to the processor unless you know it is not waiting due to I/O or 
enqueue contention. 

Once the causes of wait time are identified, initial solutions involve increasing 
access to the resources causing the delay via specifying preference of the work in 
question over other work in the system (that is, workload management). Means 
of doing this include dispatching priority, the IPS performance objective, and, for 
batch/IMS, the number and structure of initiators/message processing regions, 
respectively. Use of such controls implies trade-offs for other work in the system. 
If such trade-offs cannot be tolerated, the direction of the analysis must shift to 
optimizing use of the resource(s) in question — or prioritizing the objectives for 
different types of work, modifying objectives, or considering additional hardware. 
Investigating the use of the major system resources — processor, I/O resources, and 
real storage — is described in Chapters II.2, II.3, and II.4, respectively. For 
example, if the processor is critical and increasing the DPRTY of the work in 
question will have an unacceptable effect on other work, you can examine the use 
of processor time to eliminate or reduce non-productive processor time — see 
Chapter II.2, "Investigating the Use of Processor Time." 

Solutions to different causes of delays can also be conflicting. For example, 
assume a TSO trivial response time problem is being investigated and two causes 
of wait time are identified: excessive paging and waiting to start. After 
investigating possible solutions to each of these problems, you focus on reducing 
the multiprogramming level of the system to reduce paging and increasing the 
maximum multiprogramming level for TSO trivial transactions to reduce their 
wait-to-start time. Other work in the system will have to be reduced to achieve the 
lower system workload level and the higher TSO workload level, which can impact 
meeting the performance objectives of the other work. (A possible benefit, 
however, is increased productive processor time due to decreased paging, which 
might offset lower multiprogramming levels by decreasing total elapsed time.) 
Again, at this point, you might have to examine the use of the critical resource 
(in this case, real storage) or consider modifying the objectives or obtaining 
additional hardware. 

Possible solutions to each of the causes of wait time are included in the bulletins 
at the end of this chapter. 
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Waiting to start 

Possible Problem: work in question is waiting to start. Investigation of and 
possible solutions to tliis problem depend on the type of work: batch, IMS, or 
ISO, as described below. 



Batch 

Investigating Problem: Wait-to-start time is available in SMF record 5 : the time the 
initiator selected the job (offset + X'27') less the time the reader recognized the 
JOB card (offset + X'16'). Determining if the wait-to-start time is excessive 
requires a judgment based on the amount of time the job spends executing and the 
objective for the job — or based on the average wait-to-start time experienced when 
jobs are receiving satisfactory turnaround time. 

Possible Solutions: Re-evaluate the number of initiators and the job classes 
assigned to each initiator. Also examine the external scheduling criteria for the jobs 
and their grouping into job classes: for example, can some of the jobs be moved to 
a separate class and that class executed during an off-shift ? Check for jobs that 
violate job class standards. 



IMS 

Investigating Problem: Input queue times for IMS transactions (equivalent to 
wait-to-start time for TSO/batch work) are reported by the IMS/VS log transaction 
analysis utility program. (See the MS/VS Utilities Reference Manual for details on 
this program.) To judge if input queue times are high, compare them to average 
input queue times observed when performance objectives are being met. Note 
that the computation of input queue time (from information on the IMS log) does 
not include possible delays at the terminal or on the terminal line. 

Possible Solutions: Several factors can contribute to IMS transactions being 
delayed on the input queue: 

• The IMS control region cannot schedule the transaction to an MPR. This can 
occur because: 

— All MPRs are active. Re-examine the number of MPRs, the classes served by 
each, and the scheduling priority of the transaction type being delayed (as 
specified in the PRTY parameter coded on the TRANSACT macro for this 
transaction type: lower the limit count if the limit priority is not being used; 
or raise the Umit priority if it is being used). 

— The control region cannot allocate space for the required control blocks (a 
non-zero return code is returned from the block-builder), due to insufficient 
virtual storage assigned to work pools (PSB, PSBW, DMB, and/or DBWP). 
Re-examine the size of the work pools. 
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Waiting to start (continued) 

• If insufficient message queue buffers are available to handle the arrival rate of 
transactions, they will be sent to the message queue data set (OSAM), whether 
or not an MPR is available to process them. This will delay the transaction by 
at least the I/O time to and from the message queue data set. 



TSO 

Investigating Problem: Examine GTF SRM trace data for the average time from 
user-ready to restore-complete SYSEVENTs. If this time exceeds approximately 
one-half second, TSO transactions are being kept out of storage. For TSO trivial 
transactions, where swapping usually occurs only once per transaction, wait-to- 
start time can also be computed from data available on the RMF workload activity 
report : 

avg. transaction service rate ^ . ^ . . 

X avg. time of ended transaction = residency time 



avg. absorption service rate 
and: 

avg. time of ended transaction — residency time = wait-to-start time 

Possible Solutions: The delay can occur for one of two reasons: 

• the swap-in is taking too long (examine the time from swap-in-start to restore- 
complete SYSEVENTs in the GTF SRM trace data). Poor swap performance is 
usually best addressed by the number and placement of page and swap data sets 
— see the Initialization and Tuning Guide for suggestions. 

• swap-in is being delayed (examine the time from user-ready to swap-in-start 
SYSEVENTs in the GTF SRM trace data). This can occur because : (1) the 
target MPL, maximum MPL, or domain weight for this domain is low compared 
to the average number of users; or (2) non-trivial transactions execute in the 
same domain as trivial transactions and are not pre-empted when new 
transactions become ready. (Note, however, that pre-emption can result in an 
excessive swap-to-transaction ratio. It is best to place trivial and non-trivial 
transactions in separate domains, as described below.) 

In the first case, adjust the MPLs and the weight for the domain. Beware, 
however, that this can adversely affect other work in the system. The target MPL 
might be low because of broader performance problems: for example, processor 
or real storage constraints. If true, it is necessary to identify the critical resource 
and examine its use. The description of step 4 in Chapter II. 1 describes how to 
focus on the major resources (system-oriented performance analysis); chapters 
II.2, II.3, and II.4 describe investigating the use of processor time, I/O resources, 
and real storage, respectively. 
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In the second case, the preferred solution is to place trivial and non-trivial 
transactions in separate domains — associate the second period of the TSO 
performance group with a different domain than the first period. If the two 
periods already specify different domains, re-examine the duration of the first 
period (as specified by the DUR parameter); it might be too long, allowing non- 
trivial transactions to execute in the first period. 

If the periods of the TSO performance group are associated with the same 
domain, an alternate solution is to ensure that transactions that enter the second 
period are pre-empted by transactions that become ready. To accomplish this, 
define the IPS performance objective such that, at zero service level, transactions 
in the second period are at a lower workload level (more than one lower) than 
transactions in the first period. On a graph, the objectives for periods 1 and 2 
should look as follows, to ensure pre-emption of transactions in period 2: 
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Workload Level 

Pre-emption of transactions in period 2 will not occur if the IPS performance 
objective results in a graph such as the following : 
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100 

Workload Level 

For details on defining IPS performance objectives, see the Initialization and 
Tuning Guide. Warning: Associating periods of the TSO performance objective 
with the same domain and using the IPS performance objective to ensure pre- 
emption can, however, result in an excessive swap-to-transaction ratio. Beware 
of the possible adverse effects of this solution, especially if real storage is a critical 
resource at your installation. 
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Access to the processor 

Possible Problem: While in storage, the work in question cannot gain sufficient 
access to the processor. 

Investigating Problem: Compare the residency time of the work in question to the 
amount of processor time used. For TSO and batch work, residency time and TCB 
processor time are available in SMF records 5 for batch jobs and records 35 for 
TSO transactions. For IMS, residency time is the time a transaction is active in an 
MPR; it is reported as "time processing" in the detailed transaction report 
produced by the IMS/VS log transaction analysis utility program. The average TCB 
processor time per message is reported on the apphcation accounting report 
produced by the IMS/VS statistical analysis utility program. 

Judgment of the processor-to-residency-time ratio should be based on several 
factors: whether the work is swappable or non-swappable; the amount of 
processor time required by the work in question; the objective of the work in 
question. It is best to judge the processor-to-residency-time ratio in light of the 
ratio usually observed when performance objectives are beirig met. As a starting 
point for swappable TSO or batch work, you might expect to see a ratio of 
approximately!: 5. 

Note that, if you decide the work is not experiencing an adequate processor- 
to-residency -time ratio, you cannot immediately assume that the work is being 
denied access to the processor. You must also consider the possibility that it is 
waiting due to I/O or enqueue contention — see bulletins 5.4.0 and 5.5.0, 
respectively. 

SIR can be used to identify those address spaces of higher dispatching priority 
that are using large amounts of processor time — in increasing the work-in- 
question's access to the processor, you will want to consider trade-offs with these 
address spaces. 



Chapter II.S: User-oriented Performance Problem Analysis 11,69 



User-oriented Problem Analysis 
• access to the processor 



BuUetin 5.2.0 March 1977 

Access to the processor (continued) 

Possible Solutions: Increase the DPRTY of the work in question. (Note, however, 
that this will affect other work in the system.) For example, in a system 
containing a DB/DC application and heavy JES activity (RJE, local printers, and so 
on), JES will require significant amounts of processor time ; as a default, JES will 
execute at a higher dispatching priority than the DB/DC application, therefore 
impacting the appUcation's access to the processor. Placing the application at a 
higher DPRTY than JES will improve its performance. 

If the work in question is in the APG mean-time-to-wait group, you can place 
a specific job above the mean-time-to-wait group to improve its access to the 
processor; or use the rotate priority for a group of jobs with heavy processor 
demands. 
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Excessive demand paging 

Possible Problem: Work in question is being delayed due to excessive demand 
(non-VIO non-swap) paging. 

Investigating Problem: In investigating if excessive demand paging is delaying the 
work in question, you first want to establish if the system in general is experiencing 
excessive paging, regardless of whether the work in question has a high paging rate. 
High demand paging can affect throughput, which affects the performance of all 
work in the system, including the work in question. Paging also "costs" in terms 
of processor time, which again can affect the work in question. The problem is 
judging if a paging rate is "high" — paging rates vary widely and your system can 
have a high rate, as compared to other installations, and good performance. Paging 
rates are high only if they are adversely affecting performance (such as costing 
excessive processor time). Chapter II.4 describes the ways in which excessive 
paging affects performance; figure 11.14 gives target values for demand paging, to 
use as a starting point if your installation is not sure how much demand paging is 
acceptable in your system. If demand paging in your system seems high, it should 
be investigated — see Bulletin 4.2.0 in chapter II.4. For IMS systems, see also the 
topic "Demand paging in IMS" in Part III. 

You can also investigate paging rates as an indication of an application's misuse 
of its own address space. Information on paging on an address space/job level is 
available after the fact via SMF and SIRBATCH, and dynamically via SIRTSO. 
(Note that, for IMS, paging rates are reported for MPRs and the control region; 
no paging statistics are available for individual IMS applications. For IMS tuning, 
however, CSA paging is the most important paging factor to investigate — not 
paging by individual applications.) 

Two numbers are of interest when investigating the potential problem of 
excessive paging for specific batch applications: page reclaims and demand page- 
ins (non-VIO, non-swap page-ins). These numbers are available on a job step 
level in SMF type 4 records and on a job level in SIR. 

The number of page reclaims can indicate an application's mis-use of its own 
address space, if it is high compared to the number of reclaims experienced by 
other address spaces. (Note, however, that a high reclaim rate is not of as great 
a concern as high demand paging rates, because of the relatively lost "cost" of 
reclaims.) 
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Excessive demand paging (continued) 

A rule-of-thumb forjudging the number of demand page-ins is that, at a minimum, 
the amount of processor time used by the job between demand page-ins should at 
least equal to the time taken to resolve a page-in. To compute the average processor 
time between demand page-ins, divide the total processor time (from SMF record 
5) by the number of demand page-ins and compare this to the average time required 
to resolve a page-in. The average time required to process a page-in can be 
computed as RCVASMQA (ASM queue length average from the RMF trace) divided 
by the total system paging rate. (Note that excessive time to satisfy a page transfer, 
as computed by this formula, can also contribute to delays of the work in question — 
see bulletin 4.4.0 in chapter II.4.) The appHcation is experiencing excessive 
paging if the average processor time between demand page-ins is less than the time 
required to resolve a page-in. A less exact test of excessive demand page-ins is a 
simple comparison of the appHcation's demand page-in rate to that of other address 
spaces in the system. 

SMF and the SU? version of SIR also provide data on whether page faults 
occurring in an address space are in its own address space or in PLPA/CSA — 
which information might be needed for determining a solution to the 
performance problem. 

Possible Solutions: If investigation of a particular application revealed excessive 
paging for that application in particular, solutions pertain to the application's use 
of its own address space — for example, repetition of a page reference pattern 
after periods of disuse. Study the program's CSECT ordering and module reference 
patterns to identify specific solutions. 
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Waiting for I/O 

Possible Problem: work in question is waiting for I/O to complete. Investigation of 
this potential problem varies for TSO or batch work and IMS work. 

TSO or Batch 

Investigating Problem: Correcting imbalances in channel and device activity on a 
system level will not necessarily correct I/O delays encountered by the work in 
question. You must investigate the specific I/O resources being used : SMF records 
type 4 (batch) and 34 (TSO) identify the devices assigned to non-spooled data sets 
and the channels used for those devices. 

Possible Solutions: Solutions to I/O delays depend on the bottleneck contributing 
to the delay — for example, operational bottlenecks due to volume mounting; 
excessive SEEK time due to poor data set placement ; and so on. Chapter II.3, 
"Investigating the Use of I/O Resources," describes bottlenecks that contribute to 
I/O delays, ways to determine if they are occurring, and suggestions for reducing 
them. Chapter 1.3, "Pre -initialization MVS Performance Factors," describes 
possible operational bottlenecks. 

IMS 

Investigating Problem: To investigate the possibility of I/O delays, start by looking 
at RMF or MF/1 channel and device reports for evidence of delay on the volumes 
of the data base used by the work in question. For example, examine queue length 
measurements and the average time to service a request on the device(s) accessed — 
see bulletin 3.1.0 in Chapter II.3 for descriptions of these measurements. Bulletin 
3.2.0 in Chapter II.3 describes measurements that might indicate specific bottle- 
necks in the path to the device. In addition, long IWAIT times in the DC Monitor 
reports might indicate possible I/O contention. 

Possible Solutions: If I/O delays can be attributed to contention for the device on 
which the data base resides, examine the placement of the data base on devices: 
consider spreading the data base across more devices; and/or ensure the placement 
of highly -used parts of the data base on different paths. I/O resource contention 
can also be reduced by reducing the number of EXCPs/SIOs — see bulletin 3.4.0 
in Chapter II.3. 
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Enqueue/data base contention 

Possible Problem: Work in question is delayed due to enqueue contention (TSO or 
batch work) or data base segment contention (IMS work). 

TSO or Batch 

Investigating Problem: The occurrence of enqueue exchange, long wait, and 
detected wait swaps (reported in swap-out counts on the RMF paging activity 
report) can indicate the presence of enqueue contention on a system level. On an 
address-space level, SIRBATCH identifies address spaces that are waiting for a 
resource, the address space holding the resource, and the name of the resource. 

Possible Solutions: The solution depends on the specific resource in contention 
and the address spaces vying for the resource. Some general suggestions to consider 
include the following: 

• Use the JCL parameter FREE=CLOSE to ensure that resources are released 
when no longer needed. 

• Assign the same job class to jobs that require the same resources and assign that 
job class to a single initiator. 

• If a specific job monopoUzes resources needed by other work in the system, run 
that job off-shift or limit the duration of its effect on other work by giving it a 
preferred dispatching priority and IPS performance objective to "hurry" it 
through the system. 

• Structure the catalogs to reduce catalog contention. 

IMS 

Investigating Problem: The IMS/VS program isolation trace report utility program 
hsts all transactions that had to wait while trying to enqueue on a data base record. 
The report includes the transaction waiting for the data base record, the transaction 
holding the record, and, if the /TRACE ALL option is specified, the elapsed time of 
the wait. 

Possible Solutions: Program isolation in IMS reduces data base contention by 
allowing enqueues on particular segments and by resolving deadlock situations. 
Therefore, solutions to contention for data base records lie primarily in the area of 
workload management; for example, scheduling transactions that require the same 
data base records to a single MPR. 
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Excessive swapping 

Possible Problem: work in question is being delayed because of swapping. 

Investigating Problem: Swapping can delay work in two ways: 

(1) The work is swapped too frequently. 

(2) Once swapped out, the work waits too long before being swapped back in. 

(1) hi the first case, examine the swapping rates of the work in question. 
Swapping rates on a performance group level are available on the RMF workload 
activity report. The number of swaps per address space is available via SIRBATCH 
or SMF (record type 4 for batch; type 34 for TSO). 

For TSO, a general target is a swap-per-transaction ratio of 1 .1 :1 ; for trivial 
TSO, 1 .0: 1 . For batch, the acceptable number of swaps per job is installation- 
dependent; in judging the number of swaps, consider factors such as the priority of 
batch work and the cost of swapping in terms of processor time. One 
rule-of-thumb is that swapping is not excessive for batch if batch swaps occur no 
more frequently than once every twenty seconds on a 158, once every eight 
seconds on a 168 (which translates to approximately 1% of processor time for 
batch swapping). 

If you judge the swapping rate of the work in question to be high, determine 
why the swaps are occurring. SIRBATCH includes swap reason codes in report 3, 
the snapshots of jobs. You can also use a GTF SRM trace to trace the swaps 
experienced by the work in question. (Although swap-out counts by type of swap 
are provided on the RMF paging activity report, they are not broken down by 
address space.) 

(2) In the second case (work in question is being delayed before being swapped 
back in), the average time in storage per swap should be at least equal to the 
amount of time out of storage per swap: 

time in storage per swap > time out of storage per swap 

Time in storage is residency time, reported in SMF record type 5 for batch, type 
35 for TSO. Time out of storage is transaction active time (also from SMF records 
5/35) less residency time. 
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Excessive swapping (continued) 

The amount of time out-of-storage can be further broken down by computing 
the actual time required to perform the swap, versus time spent on auxiliary 
storage. Via a GTF SRM trace, compute the time from quiesce-start to swap- 
complete SYSEVENTs (swap-outs) and from swap-in-start to restore-complete 
SYSEVENTs (swap-ins) for swaps of the work in question; and take their average. 

Possible Solutions: The solution to minimizing swaps depends on the type of swap 
being experienced. See Bulletin 4.1.0 in Chapter II.4. 

If work is being delayed before being swapped back in, the solution might also 
depend on the type of swap — for example, on a long or detected wait swap, the 
delay might be caused because the address space is enqueued on a resource in use 
by another address space. The delay can also be caused by an inappropriate 
definition of the IPS performance objective for the work in question (for example, 
improper domain weights). For information on these factors, see the Initialization 
and Tuning Guide. 

Some appUcations (such as DB/DC applications) should be executed as non- 
swappable. For IMS, it is recommended that the message processing regions be 
made nonswappable. (It is a requirement that the IMS control region be made 
nonswappable.) 
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Increasing the SAM buffer pool size via BLKSIZE and/or BUFNO will help reduce 
EXCP rates. The EXCP/IOS path length in MVS has increased substantially from 
MVT. The number of times this code is executed, however, can be reduced either by 
increasing blocksizes or by chaining requests together. Since chained scheduling 
(OPTCD=C) is now the default in most cases, increasing the number of buffers 
usually increases the number of requests that are chained together. 

Although there is a trade off of storage for processor cycles with this suggestion, 
large buffer pools in MVS no longer occupy a major part of the potential user 
region. Since an individual buffer is fixed only while I/O is being performed to it, 
the system can automatically adjust real storage requirements based on the actual 
data rate. However, large buffer pools in an I/0-limited process can lead to large 
real storage requirements and long periods of channel monopolization. 

Use an analysis of SMF records 14/15 to focus on data sets that have high EXCP 
counts. Consider the following: 



Unit Record Devices: QSAM is preferable to BSAM for all options. For output, 
use control characters rather than the CNTRL macro. For printers, specify 
OPTCD=C and do not use PRTOV: they are incompatible. 



Tape: Use QSAM unless NOTE/POINT or BSP is needed. With QSAM (or a user- 
provided I/O overlap) avoid RECFM=U and choose a multiple of 4096 for 
BLKSIZE. For QSAM programs that are limited by tape I/O speeds, you can 
assume that half of the buffer pool is fixed at a time, and that the channel is locked 
long enough to transmit the corresponding amount of data. 



DASD: Allocate data sets by cylinders and use full or half-track blocking if most 
processing is sequential. Allocating data sets by cylinders reduces overhead 
associated with testing if end-of-extent has been reached, since the test is only 
performed at the end of each cylinder rather than at the end of each track. 

Where possible, use QSAM rather than BSAM to provide full automatic I/O over- 
lap and complete BLKSIZE/BUFNO flexibility. Increasing BSAM NCP will have 
no effect unless the application program provides I/O overlap and can adjust to an 
increased number of outstanding requests. Increasing QSAM BUFNO can reduce 
both EXCP/IOS time and total elapsed time to a fraction of their former value. 
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Avoid track overflow or undefined length records (RECFM=U or T). To reduce 
channel busy time, use RECFM=FBS when creating physical sequential data sets. 
With PBS, the incorrect-length flag in the Read Data CCW indicates the last block 
in the data set. Therefore, an EXCP need not be issued to read the EOF mark. 
RECFM=FBS cannot be used where short blocks might be imbedded within the 
data. 

You can achieve minimum EXCP overhead by setting BUFNO (or NCP) to twice 
the number of blocks per allocation unit (track or cylinder). Note, however, that 
this large a specification is not usually practical. With cylinder allocation and an 
apphcation that is DASD limited, this approach will lead to channel lockouts, 
including block multiplexor channels, of up to 1/2 second. It will also force 
concurrent page fixes for up.to 60 pages, and as much as 2K of SQA per data set. 
The high real frame requiremens combined with the channel lockout could 
seriously degrade the system if the same channel is used for paging. 



Procedure Libraries: Determine the size (the number of 80-byte records) of each 
procedure in the procedure library. Choose a blocksize that will allow a high 
percentage (85 - 90%) of the procedures to be totally read into storage in one 
EXCP. A 4K minimum blocksize is suggested. If reading data of few record 
members, override the chained-scheduling default. 



Program Libraries: Program libraries should be examined for modules that are 
link-edited with the 'DC attribute. If the load modules are frequently used, 
link-edit them again to remove the 'DC attribute. This will minimize the processor 
time required to fetch those modules by reducing PCI handling. (Note, however, 
that this might result in an increase in elapsed time, due to missed connections, if 
there are multiple RLDs per text block.) PCIs can also be reduced by increasing 
program libraries to full track blocking. The linkage editor has specific require- 
ments when building the libraries at full track. See OS/VS Linkage Editor and 
Loader. 
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Optional Channel Paths: The current implementation of the channel scheduler for 
MVS does not provide for rotation of the physical channel paths within logical 
channels. Thus, if the OPTCHAN keyword is specified on the lODEVICE macro, 
making optional channel paths available to devices, I/O will always be started on 
the primary path first (the logical channel with the lower channel number). 
Only if the primary channel path is busy will I/O be tried on the secondary channel 
path (the higher channel number). The logical channel with the lower channel 
number in an optional channel pair will thus tend to be busy more often. Although 
this does not significantly impact the performance of tape of non-RPS DASD 
devices, it does affect RPS DASD. The RPS operation must reconnect with the 
channel that started the operation; the imbalance of channel activity, however, 
increases the probability that it will not be able to reconnect. 

You can increase the balance between optional and primary channels by the way 
in which you configure I/O resources. When devices that have only one I/O path 
are placed on a channel for which the OPTCHAN option is specified (that is, a 
channel which is one of several paths to another device), configure them on the 
logical channel with the higher channel number. This helps achieve balance since 
the lower channel will probably be overloaded. However, if channels are selector 
channels, or if devices are in BURST mode (for example, tape), which makes 
multiplexor channels appear as selector channels, then devices should be placed on 
the logical channel with the lower channel number instead. This is because the 
lower channel gets driven first. By the time the logical channel with the higher 
channel number gets driven, that channel might be busy. It is best, for selector- 
mode devices, to avoid configurations that have multiple logical channels sharing a 
common physical channel. 
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This topic presents some guidelines forjudging the reasonableness of the MVS 
hardware configuration. The recommendations for the basic hardware resources are 
"rules-of-thumb" forjudging the adequacy of your configuration; the formulas do 
not take into account tuning options and workload balancing. The likelihood of 
meeting performance objectives with current hardware resources depends on how 
well tuned the prior system was, flexibility in workload balancing, the size of the 
discrepancy between what is recommended and the actual resource (if a 
discrepancy exists), and similar factors. Although these guidelines, therefore, do 
not provide a firm answer to the question of adequate resources, they are useful as 
foresight for sizing the tuning effort and for predicting the likelihood of success 
with current hardware. 



Processor 

Processor utilization in MVS might be higher than in previous systems, primarily 
because of two factors: 

• Reduced contention for serially usable resources in MVS (such as allocation 
resource serialization, fixed regions of real storage, and certain system data sets). 
This reduces processor wait time. 

• Increased supervisor path lengths to support the additional function provided by 
MVS. Depending on the usage of various supervisor functions by a given work- 
load, this might or might not cause an increase in processor time required to 
service that workload. 

Consider the following guidelines when converting from MVT or SVS or from two 
MVS uniprocessors to an MP: 



MVT: If converting to MVS from MVT, processor upgrade might be indicated if 
EXCP rates on the MVT system exceeded 100 per second on a 158, or 300 per 
second on a 168, and the major I/O files used by the workload cannot be reblocked 
to reduce the number of EXCPs required to read the same amount of data. 
Processor utilization in MVS can be estimated for most MVT environments with the 
following formula, based on hardware monitor measurements: 

MVS processor:^ MVT problem state + (2 x MVT supervisor state) 
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SVS: Several measurements have shown SVS and MVS processor utilization to be 
approximately the same to support similar workloads, but no measurements have 
been made in heavy I/O environments. 



MVS UPs to MP: The processor is potentially a critical resource when converting 
from two MVS uniprocessors to an MP environment if total average utiHzation 
from both MVS UPs is greater than approximately 160% (as measured by MF/1 , 
RMF, or a hardware monitor). 



I/O Resources 

Existing I/O resource configurations (channels, control units, and devices) should 
be, at the least, maintained on the MVS system. MVS systems frequently 
experience increased channel utilization compared to MVT and SVS. If the utiHza- 
tion of channels to existing RPS devices exceeds approximately 35% (measured by 
OSPTl or a hardware monitor) on the pre-MVS system and response-oriented 
applications are to be supported on the MVS system, consider additional block 
multiplexor channels and control units for the MVS system. I/O resources should 
also be upgraded if growth or new applications are planned. 



Real Storage 

The rules-of-thumb in figure III.l can be used to roughly estimate the amount of 
real storage required for MVS; figure III.2 is a worksheet for your estimates. 



III.8 0S/VS2 MVS Performance Notebook 



Configuration 
• mles-of-thumb 



Configuration Rules-of-Thumb (continued) March 1977 



Basic System Requirements: 

Nucleus 400-450K 

SQA 1 50-200K + 3K per address space 

Master scheduler 20-60K 

LPA and CSA 

2 meg environment 600— 900K 

3 meg environment 900— 1200K 

4 meg environment 1 200-1 500K 

Batch Requirements: 

Assume all active initiators are swapped in. 

Configure real storage as greater than or equal to one-half the sum of the 
REGION sizes specified on JOB cards. (This includes approximately 20K 
for LSQA for each swapped-in address space.) 

JES2 requires a range of 100K to the region size specified for it. 

JESS global requires a range from the ASP region size to 1,5 times the ASP 
region size. 

JESS local requires a range of 80K to 120K. 

TSO Requirements: 

TCAM EP (emulator program) requires 150-250K (150K for approximately 
10 terminals; 250K for 25-S5 terminals). 

VTAM requires 450-650K. 

Swapped-in users = 20% of logged-on users. 

Real storage required = 70K per swapped-in user. (This includes 
approximately 20K of LSQA per swapped-in user.) 

DB/DC and On-Line Requirements: 

Until page reference patterns are understood, configure real storage equal 
to the "region size." 



Figure III.l Rules-of-Thumb for Real Storage Requirements 
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Installation 



Date 



System Profile 
Processor 



Storage 

Nucleus 

SQA 
Base 



TSO Active: Avg. 
Max 



3K X No. address spaces 
Master Scheduler 
LPA + CSA 
JES2 
TCAM 

TSO ( in) 

BATCH ( in) 

DB/DC 
Available 

Total 



Batch I nit. 
DB/DC 



Figure III.2 Worksheet for Estimating Real Storage Requirements 
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The following are miscellaneous hints on MVS hardware configurations. See also 
the preceding topic, rules-of-thumb for hardware configurations. 



3270s and Tapes: Do not place local 3270s on the same selector or block multi- 
plexor channel as tapes. Tapes should not be attached to a block multiplexor 
channel at all unless they are the only online device type on the channel, since they 
tie up the channel for the entire I/O operation. lOS will not service 3270 attention 
interrupts as long as tape I/O operations are active on the channel and additional 
requests are queued to be started when the current operation completes. 
Therefore, placing local 3270s and tapes on the same channel results in long 
response time delays for the 3270. 



2305 vs. Additional Real Storage: If excessive swapping and demand paging are 
degrading performance in your system, consider the following: 

• Adding a megabyte of real storage to reduce the amount of swapping and paging 
necessary seems to help batch environments of three megabytes or less. 

• Adding a 2305 device to increase the speed of swapping and paging activity 
seems to help larger systems or systems with high TSO activity. (No amount of 
real storage will reduce the once-in/once-out swapping activity of TSO users for 
each transaction. However, additional real storage can result in fewer pages 
actually being transferred to auxiliary storage: more pages will be reclaimed.) 
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Changes to the SRM in SU7 can resuh in higher demand paging rates for IMS, 
whicli can in turn impact IMS response times in IMS-mixed environments. The 
following paragraphs highlight SU7 changes that affect demand paging and suggest 
ways to prevent excessive IMS paging in IMS-mixed environments. This write-up 
assumes knowledge of the SU7 SRM, including the indicators SRM uses to judge 
system contention, the page-stealing algorithm, and the concept of domains and 
MPLs. 

There are two basic changes in the design of SRM that are significant to IMS 
paging: 

• Address spaces are not swapped out to resolve storage constraints until the 
indicators of system contention are exceeded. Until the indicators are exceeded, 
storage shortages are resolved by demand stealing on a system-wide basis. As a 
result, the SU7 SRM might allow an environment to exist that is too storage- 
constrained for good online response times because its thresholds for recognizing 
system contention might be too severe for an online system. 

• To determine which frames are eligible for stealing, the page-stealing algorithm 
adjusts every frame's unreferenced interval count (UIC) based on whether the 
frame was referenced in the preceding interval. Unlike the former SRM, there is 
no updating based on an address space's processor time, or any difference 
between swappable and non-swappable address spaces or between private and 
common areas. As a result, the frames likely to be stolen in systems with SU7 
installed might be different from prior systems: address spaces that were 
insulated from stealing in pre-SU7 systems — such as IMS — can suffer more 
severe stealing; areas of CSA and LPA with infrequent reference patterns — 
such as areas used by IMS — are no longer protected by the common bound 
algorithm. 

There are two approaches to keep IMS from suffering excessive paging in 
systems with SU7 installed : 

(1) by controlling other work in the system so that IMS receives sufficient 
resources. 

(2) by changing the thresholds (the "happy values") by which SRM recognizes 
system contention and responds by lowering the multiprogramming level 
in the system. 

Each of these approaches is described in more detail below. 
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Controlling non-IMS Work 

If the SU7 SRM allows processor or storage utilization to be too high for IMS, you 
can limit the amount of work in the system by setting an appropriate maximum 
MPL for the TSO or batch domains - in the IPS or dynamically via the SET 
DOMAIN operator command. For batch, you might be able to achieve the same 
result simply by limiting the number of initiators. This should be effective if the 
batch workload is stable and can be controlled via installation scheduling — that is, 
if the amount of storage or processor time consumed by batch is directly related to 
the batch multiprogramming level. 



Modifying the "Happy Values" 

You might investigate changing (via superzap) the "happy values" for the indicators 
by which SRM recognizes system contention: processor utilization, ASM queue 
length, paging rate, or highest UIC in the system. Consider the following implica- 
tions of comparing UIC count and ASM queue length "happy values" in IMS/batch 
versus TSO/batch systems: 

UIC Count: Because of the structure of IMS (the IMS control region being 
basically one task that does its own internal subtasking) and the fact that it is 
terminal-driven, IMS pages tend to be unreferenced longer than TSO or batch pages. 
The default mean UIC happy value is 3. IMS/batch systems will tend to have a 
larger number of unreferenced pages incremented to 3 in a given time period than 
TSO/batch systems. As a result, an IMS/batch system will tend to have a higher 
paging rate with a UIC range of 2-4 than a TSO/batch system with the same range. 
This might indicate that the UIC count happy values should be increased in 
IMS/batch systems. 



ASM Queue Length: The happy range for ASM queue length is 7-10. However, 
this includes swap pages as well as demand pages; and TSO/batch systems generally 
experience more swapping than IMS/batch systems. If half the ASM queue length 
in a TSO batch system is swap pages (which would not be unusual), the happy 
range of 7-10 is actually a happy range of 3.5 - 5 demand pages. In an IMS/batch 
system, with less swapping, the actual demand paging range is closer to 7-10. This 
might indicate that the ASM queue length happy range should be lowered in an 
IMS/batch system. 
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Dispatching priorities, which are a major means of controlling the access of users to 
the processor, should be based on the workload priorities at your installation. The 
Initialization and Tuning Guide describes dispatching priority control. Following 
are some general guidelines to consider : 

• Critical tasks and I/0-bound tasks should be given high priorities. 

• In general, all address spaces except the master scheduler, IMS, JES2, and TCAM 
should be controlled by the SRM via the APG keyword in the IPS. The SRM 
attempts to increase system throughput by dynamically adjusting the 
dispatching priority of address spaces within the mean-time-to-wait (MTTW) 
group of the APG. Use of the MTTW group, therefore, prevents domination of 
the processor by single users. However, a very heavy processor-bound job or 
TSO session could be hurt by being in the MTTW group, since the SRM would 
place it near the bottom of the MTTW group with a low priority. TSO users 
should be placed outside the MTTW group with a higher priority than batch. 
Batch jobs should usually be run in the MTTW group. 

• For TSO work assigned to the APG, you should associate different periods 
within the TSO IPS performance objectives with different fixed priorities within 
the APG. During the first period, when trivial commands are executing, assign a 
high APG priority (above the mean-time-to-wait (MTTW) subgroup of APG 
priorities). During successive periods, when medium TSO work is executing, 
TSO should be given a lower fixed APG priority. During final periods when 
heavy TSO work is executing, and the TSO user is most demanding on system 
resources, the work should be moved into the MTTW APG priority and be run 
like a batch job. 
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Changes to the SRM m SU7 can resuh in higher demand paging rates for IMS, 
which can in turn impact IMS response times in IMS-mixed environments. The 
following paragraphs highlight SU7 changes that affect demand paging and suggest 
ways to prevent excessive IMS paging in IMS-mixed environments. This write-up 
assumes knowledge of the SU7 SRM, including the indicators SRM uses to judge 
system contention, the page-stealing algorithm, and the concept of domains and 
MPLs. 

There are two basic changes in the design of SRM that are significant to IMS 
paging: 

• Address spaces are not swapped out to resolve storage constraints until the 
indicators of system contention are exceeded. Until the indicators are exceeded, 
storage shortages are resolved by demand stealing on a system-wide basis. As a 
result, the SU7 SRM might allow an environment to exist that is too storage- 
constrained for good onhne response times because its thresholds for recognizing 
system contention might be too severe for an online system. 

• To determine which frames are eligible for stealing, the page-stealing algorithm 
adjusts every frame's unreferenced interval count (UIC) based on whether the 
frame was referenced in the preceding interval. Unlike the former SRM, there is 
no updating based on an address space's processor time, or any difference 
between swappable and non-swappable address spaces or between private and 
common areas. As a result, the frames likely to be stolen in systems with SU7 
installed might be different from prior systems: address spaces that were 
insulated from stealing in pre-SU7 systems — such as IMS — can suffer more 
severe stealing; areas of CSA and LPA with infrequent reference patterns - 
such as areas used by IMS — are no longer protected by the common bound 
algorithm. 

There are two approaches to keep IMS from suffering excessive paging in 
systems with SU7 installed : 

(1) by controlling other work in the system so that IMS receives sufficient 
resources. 

(2) by changing the thresholds (the "happy values") by which SRM recognizes 
system contention and responds by lowering the multiprogramming level 
in the system. 

Each of these approaches is described in more detail below. 
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Controlling non-IMS Work 

If the SU7 SRM allows processor or storage utiUzation to be too high for IMS, you 
can limit the amount of work in the system by setting an appropriate maximum 
MPL for the TSO or batch domains - in the IPS or dynamically via the SET 
DOMAIN operator command. For batch, you might be able to achieve the same 
result simply by limiting the number of initiators. This should be effective if the 
batch workload is stable and can be controlled via installation scheduling — that is, 
if the amount of storage or processor time consumed by batch is directly related to 
the batch multiprogramming level. 



Modifying the "Happy Values" 

You might investigate changing (via superzap) the "happy values" for the indicators 
by which SRM recognizes system contention: processor utilization, ASM queue 
length, paging rate, or highest UIC in the system. Consider the following implica- 
tions of comparing UIC count and ASM queue length "happy values" in IMS/batch 
versus TSO/batch systems: 

UIC Count: Because of the structure of IMS (the IMS control region being 
basically one task that does its own internal subtasking) and the fact that it is 
terminal-driven, IMS pages tend to be unreferenced longer than TSO or batch pages. 
The default mean UIC happy value is 3 . IMS/batch systems will tend to have a 
larger number of unreferenced pages incremented to 3 in a given time period than 
TSO/batch systems. As a result, an IMS/batch system will tend to have a higher 
paging rate with a UIC range of 2-4 than a TSO/batch system with the same range. 
This might indicate that the UIC count happy values should be increased in 
IMS/batch systems. 



ASM Queue Length: The happy range for ASM queue length is 7-10. However, 
this includes swap pages as well as demand pages; and TSO/batch systems generally 
experience more swapping than IMS/batch systems. If half the ASM queue length 
in a TSO batch system is swap pages (which would not be unusual), the happy 
range of 7-10 is actually a happy range of 3.5 - 5 demand pages. In an IMS/batch 
system, with less swapping, the actual demand paging range is closer to 7-10. This 
might indicate that the ASM queue length happy range should be lowered in an 
IMS/batch system. 
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Dispatching priorities, which are a major means of controUing the access of users to 
the processor, should be based on the workload priorities at your installation. The 
Initialization and Tuning Guide describes dispatching priority control. Following 
are some general guidelines to consider : 

• Critical tasks and I/0-bound tasks should be given high priorities. 

• In general, all address spaces except the master scheduler, IMS, JES2, and TCAM 
should be controlled by the SRM via the APG keyword in the IPS. The SRM 
attempts to increase system throughput by dynamically adjusting the 
dispatching priority of address spaces within the mean-time-to-wait (MTTW) 
group of the APG. Use of the MTTW group, therefore, prevents domination of 
the processor by single users. However, a very heavy processor-bound job or 
TSO session could be hurt by being in the MTTW group, since the SRM would 
place it near the bottom of the MTTW group with a low priority. TSO users 
should be placed outside the MTTW group with a higher priority than batch. 
Batch jobs should usually be run in the MTTW group. 

• For TSO work assigned to the APG, you should associate different periods 
within the TSO IPS performance objectives with different fixed priorities within 
the APG. During the first period, when trivial commands are executing, assign a 
high APG priority (above the mean-time-to-wait (MTTW) subgroup of APG 
priorities). During successive periods, when medium TSO work is executing, 
TSO should be given a lower fixed APG priority. During final periods when 
heavy TSO work is executing, and the TSO user is most demanding on system 
resources, the work should be moved into the MTTW APG priority and be run 
like a batch job. 
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In addition to the definition of domains themselves (that is, choosing the number 
of domains and the work to be associated with each), you have two controls over 
domains: specifying multiprogramming levels (MPLs) and domain weights. 
Properly selecting the values of these controls is very important, as they will 
prevent the SRM from over-reacting to changing activity levels throughout the day. 
The Initialization and Tuning Guide describes these controls (called domain 
constraints). Following are some suggestions: 

• When modifying domain controls to address a problem of real-storage over- 
commitment (too many address spaces competing for available pageable 
storage), try to solve the problem first by changing the MPLs instead of the 
domain weights. Results of MPL changes are easier to observe and therefore the 
effectiveness of your solution can be more easily judged. 

• For TSO domains, set an upper MPL boundary to limit the number of 
concurrent users in storage at once. Such a boundary can provide more 
consistent resource use by restricting transient overloading. The actual 
maximum MPL value to be specified in the CNSTR keyword of the IPS is 
determined by experimentation and should be equal to the lowest value that 
will still provide adequate response time. Response time and resource conten- 
tion can be analyzed on a domain basis via the RMF workload activity report, 
and by address space via SMF reports. A guide to setting the number can be: 

0. 1 X the largest number of terminals expected to be active concurrently 

• Initially, for batch work, consider setting a minimum MPL of or 1 and a 
maximum MPL of 255, thereby giving SRM full control of the domain. By 
doing this, you allow the SRM to react to changes in system utilization and to 
raise or lower the target MPL as needed to increase system throughput. 
However, after analyzing RMF reports to determine the reasons why the SRM is 
adjusting the MPL, you might find that a redefinition of the MPLs would be 
beneficial. Remember that allowing too many address spaces of a domain in 
real storage at one time by setting the minimum MPL too high causes excessive 
paging. However, setting maximum MPLs too low will waste valuable system 
resources. Both result in reduced system throughput. 



Part III: Performance Hints III. 17 



III. 18 0S/yS2 MVS Perfonnance Notebook 



JES2 



0S/VS2 MVS Performance Notebook 
Part III 



JES2 Hints 



March 1977 



JES2 initialization parameters are among the major factors that determine the level 
of performance for JES2. Careful specification of several of them will help 
optimize the use of the system resources and thus maximize performance. 
Further information regarding the hints that follow, as well as additional factors 
and suggestions that can improve JES2 performance, are documented in 0S/VS2 
MVS System Programming Library: JES2. 



JES2 Buffer Size: Choose a buffer size for each device that is as close as possible 
to 4K or 2K. EXCPs and storage are saved when an integer multiple of buffers can 
fit on a track. Figure III.3 lists suggested buffer sizes for various system sizes and 
devices. 



System Size 




Buffer Size 




in Megabytes 


DASD 


In Bytes 


&BUFSIZE= 


>3 


3330, 3340 


4096 


4008 


>3 


3350 


3664 


3576 


3 


2314 


2048 


1960 


2 


2314,3330,3340 


2048 


1960 


2 


3350 


1954 


1866 


<2 


2314,3330,3340,3350 


1018 


930 



Figure IIL3 Suggested Buffer Sizes for JES2 Buffers 



&NUMTGV: Calculate the number of track groups for each device, rather than 
allowing &NUMTGV to default. Values used for &NUMTGV should be high since 
a small value tends to waste space, can lead to excessive seeks, and could possibly 
limit the number of total jobs to much less than the value of &MAXJOBS. Figure 
III.4 lists suggested basic values for &NUMTGV for different devices; Figure III.5 
lists suggested values for different combinations of devices. 





Number of Tracl<s In 


Space of a Logical 




Device 


Logical Cylinder 


Cylinder 


&NUMTGV= 


2314 


10 


73K 


400 


3330-1 


6 


78K 


1279 


3330-1 1 


8 


104K 


1919 


3340-35 


12 


100K 


348 


3340-70 


12 


100K 


696 


3350 


6 


114K 


2775 



Figure 111.4 Suggested Values for &NUMTGV for Different Devices 
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Devices 


Number of Tracks in the Logical Cylinder 


&NUMTGV= 


2314 


3330-1 


3330-11 


3340-35 


3340-70 


3350 


2314+3330-1 


5 


9 










800 


3330-1 -H 3330-1 1 




4 


8 








1919 


3340-35 + 3340-70 








6 


12 




696 


2314 + 3340-35 


10 






10 






400 


2314 + 3340-70 


5 








10 




800 


3330-1 +3340-35 




11 




6 






696 


3330-1 + 3340-70 




11 






12 




696 


3330-1 1 + 3340-35 






16 


6 






696 


3330-1 1 + 3340-70 






11 




6 




1392 


2314+3330 + 3340 


5 


9 


19 


10 


10 




800 


3330-1 1 + 3350 






6 






6 


2380 


Notes: 


1 . Because of the great difference in device characteristics, try not to mix 3350 and 
2314; 3350 and 3330-1 ; or 3330-1 1 and 2314. 


2. For the 3330s, try to choose a value of &NUMTGV that causes few oscillations 
(splitting of the allocation unit onto two cylinders). For example, 1096 for the 
3330-1 is not a good choice. 



Figure III.5 Suggested Values for &NUMTGV for Different Combinations of Devices 



&NUMDA: If spool activity is high, consider adding another spool volume by 
increasing the value of the &NUMDA parameter to one more than you think you 
actually need. This will split the load between spool volumes and thus improve 
performance. 



&DEBUG: The &DEBUG generation parameter should not be used in a production 
environment since it adds extra instructions to the JES2 code unnecessarily. 



Job Joumaling and SMF Records: To reduce overhead, ask for job journaling 
explicitly only when needed and turn off unused JES2 -generated SMF records 
and exits (unless they are needed for analysis). Specify the following 
subparameters in the job class initialization parameters (&STC, «feTSU, and &x): 

• NOJOURN (not applicable if your installation has SU3 installed) 

• NOTYPE26 

• NOUJP 

• NOUSO 

• NOLOG 

• N0TYPE6. 
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Sn Initialization Parameter: Do not specify more than one Sn initiahzation 
parameter in a single system configuration. Overhead will be reduced since JES2 
will need to access only the checkpoint records on a warm start ; that is, according 
to the needs of the single-mode system. 



QCONTROL in MAS Configuration: Balance workload in a Multi-Access Spool 
(MAS) configuration by specifying the QCONTROL initialization parameter. The 
QCONTROL parameter allows each system adequate access to the shared job 
queue. Among similar JES2 systems, the defaults for the QCONTROL 
subparameters will provide this balance. However, in configurations that consist of 
unequal systems, time values should favor the slower systems. For example, in a 
168/158 system complex, the following QCONTROL values will prevent the 158 
from being blocked: 





158 


168 


HOLD= 


150 


100 


MINDORM= 


100 


150 


MAXDORM= 


500 


700 



MAXJOBS: The value specified in MAXJOBS refers to all work in the system 
under control of JES2, not only jobs in execution; for example, it includes jobs 
on the queue, idle initiators, and so on. Therefore, be sure not to specify too small 
a value for MAXJOBS. 
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Data Set Placement: Plan the allocation of data sets on each volume to minimize 
the average distance travelled per seek. Rarely used data sets should be placed at 
the very beginning or end of the volume, but not under fixed heads. This 
effectively reduces the size of the active volume. The VTOC should generally be 
surrounded by the most active data sets and placed 1/3 to 1/2 of the way into the 
volume. (Examples of heavily used data sets are catalogs and page data sets; 
LINKLIB data sets are moderately used.) An exception to this guideline is on fixed 
head units in a 3340-F or 3350. More seeks are saved on these devices if the VTOC 
is placed within the fix area. Another exception is on volumes such as SYSRES, 
which are normally accessed by many tasks. On these volumes, the VTOC might be 
rarely used, and can therefore be considered as just another data set; that is, place- 
ment at the end of the pack produces the greatest reduction in seek time. Keep in 
mind that all data sets should be allocated on cylinder boundaries, where possible. 



Module Ordering Within PDSs: Re-order modules within critical partitioned data 
sets, such as heavily used libraries, according to their frequency of use. This will 
eliminate seeks within the data set. The following technique can be used to 
re-order a partitioned data set: 

1 . Use the SEEKANAL measurement tool or GTF SIO trace records to determine 
the most frequently used members in the data set. 

2. Arrange these modules in groups that will fit within cylinder boundaries. (As it 
is difficult to calculate the cylinder boundary exactly, prepare lEBCOPY COPY 
and SELECT statements for each module.) 

3. Using the lEBCOPY utility, copy the modules in the preferred order into a 
temporary data set. 

4. Using lEBCOPY without the SELECT statement, copy the whole temporary 
data set into the new permanent data set. No sorting will take place. 

5. Keep the list in PARMLIB for further reference and updates. 
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Shared DASD, particularly between loosely-coupled systems, can cause generalized, 
often unnoticed, performance problems. Reducing hardware costs by poolmg 
DASD might actually increase costs by reducing system performance. Although 
processor utilization might not decrease noticeably (due to additional processor 
cycles in more frequent task switching), shared DASD can cause application 
programs to execute more slowly. This can be especially serious in response- 
oriented systems, such as IMS, CICS, or TSO environments. System throughput 
can actually degrade by as much as 10-15% due to short duration lockouts and to 
device or control unit interference and contention. 

It is recommended that you share as few DASD strings as possible, consistent 
with reasonable back-up. When implementing shared DASD, consider the 
following suggestions and potential problem areas. 



Operational Problems in Disk Pack Management (JES2 only): Considerable human 
intervention might be necessary to keep track of the status of the shared devices. 
For example, the operators should initiate all shared volume mounting and ■ 
demounting operations, but these operations must be done in parallel on all sharing 
systems. Valid combinations of volume mount characteristics for all sharing 
systems must also be maintained. Device availability problems can easily cause jobs 
to abend or can waste available spindle time. Difficulties like these tend to increase 
substantially with each additional loosely-coupled system, since requirements for 
inter-operator coordination and communication become increasingly complex. 
Keep in mind that the operating procedures at your installation might have to be 
revised as your uses of shared facilities evolve. 



String Switching and Isolating Shared DASD: Consider string switching (the 
facility available with the 3333 storage control module) for all shared DASD 
strings. This will provide for maximum availability by backing up the DASD 
control units in the event of a hardware failure, and might benefit performance 
by reducing control unit contention. A disadvantage of string switching, however, 
is that SIO operations might return false device busy indications. Since the system 
cannot distinguish between the devices being busy or the storage control module 
(string) being busy, it always assumes that the device is busy. All I/O will be 
queued instead of being scheduled to the optional channel path. A more efficient 
use of the resources is gained by installing string switching, but inactivating it via 
the VARY OFFLINE command during normal system operation. This technique 
will make maximum use of the OPTCHAN feature in MVS. 



Part III: Performance Hints UI.25 



Shared DASD 



Shared DASD Hints (contmued) March 1977 

Volumes that must be shared, such as catalogs and program library packs, should 
be isolated from all other types of volumes and placed on a single string of devices 
where possible. This will isolate shared control unit interference to a single string 
of DASD. However, if the string is heavily used, then all systems sharing the DASD 
might suffer. Another problem exists if a control unit containing shared system 
files becomes inoperable. It will be necessary to quiesce and IPL all attached 
systems in order to relocate these essential files on a working control unit. This has 
the effect of multiplying the number of IPLs associated with hardware failure times 
the number of processors. This cost can be very high. However, string switching 
this DASD string, combined witji optional channel paths into both control units, 
will provide for preventive maintenance without causing multiple system IPLs, and 
will reduce control unit contention as well. 



Library Maintenance: Library maintenance (such as link edits) to production 
libraries should be scheduled in light processing hours, if possible, and should 
always be scheduled on a single processor. This will minimize performance 
problems caused by contention. And, by eliminating reserve/release interference 
during busy periods, the possibility of multi-system lock outs will also be 
minimized. 
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Bulletin 2.3.0, "SVC Analysis," in Part II describes how to identify SVCs that are 
"costly" in terms of processor time and whose "cost" can be decreased — by 
decreasing the frequency of the SVC or by reducing the SVCs processor time. 
Following are specific hints. 



Authorizing Programs: If a program can be authorized, use APF to authorize it. 
This will cause validity checking to be bypassed in SVCs (for example, in 
ENQ/DEQ, GETMAIN/FREEMAIN, WAIT/POST), therefore shortening their 
path lengths and processor time. 



STIMER and TTIMER (SVCs 47 and 46; modules IGC0004F and IGC0004G): 

You can reduce the instruction length — and, therefore, the processor time — of 
these SVCs by placing them in the fixed LPA. STIMER and TTIMER fix them- 
selves before getting a spin lock. However, when placed in the fixed LPA, they 
recognize prior fixing and branch around their FIX and FREE code. This cuts 
the number of instructions required to execute each macro approximately in half 
(from approximately 2000 instructions per call to approximately 1000 
instructions). 

Beforeimplementing this hint, consider the following factors: (l)the 
criticalness of real storage in your system (increases in fixed LPA will decrease the 
amount of pageable storage available); and (2) how frequently the SVC is issued. 
(SVC frequencies are reported by GTFSVC, lUP # 5796-PGE.) If the SVC is issued 
infrequently — for example, five or fewer times per second — the savings in 
processor time will not be significant. 



ENQ/DEQ: If you have shared DASD in your system, ensure that only the devices 
to be shared are generated as shared during sysgen. The path length of ENQ/DEQ 
increases significantly for devices generated as shared. 
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TCAM Buffers: Consider the following suggestions when defining TCAM buffers: 

• The amount of storage allocated for TCAM's buffers should fit into an integer 
multiple of a page. This is because storage for the buffer unit pool is obtained 
in page multiples. Buffers, which are fixed in real storage, do not cross these 
page boundaries. Therefore, it is important that you optimize the amount of 
storage used (the number of buffers x buffer unit size) to prevent the waste of 
unallocated space in the pages or the use of too much real storage. You can 
either utilize the pages fully (by increasing the size of the buffer unit), or 
minimize the number of pages that are fixed (by decreasing the number of 
buffers). Generally, the number of buffers actually used will decrease as the size 
of the buffer unit increases. Therefore, the most effective way to use the pages 
is usually to increase the buffer unit size via the UNITSIZE parameter on the 
TSOMCP macro instruction or the KEYLEN parameter on the INTRO macro 
instruction. This, however, depends on the characteristics of your particular 
TCAM installation; for example: 

— The typical peak number of terminal users. 

— The nature of user activity (amount of terminal I/O, length of messages, and 
soon). The amount of terminal I/O can be measured using GTF. In general, 
for TCAM EP, there should be at least 1.7 buffers per line; NCP requires 
fewer. 

— The type of terminals. Larger buffers are recommended for display stations. 

Specify unit size in multiples of 8 plus 4 (for example, 36, 44, and so on): 
TCAM adds 12 bytes to each unit for CCWs and control information, and units 
begin on a doubleword boundary. 

• TCAM 2701/2/3 or 3705 EP only: Since a larger buffer size causes additional 
buffers to be needed less frequently, PCI processing (which allocates buffers 
dynamically when the initial buffer specification is insufficient) is also reduced. 
However, you might want to eliminate PCI interrupts completely by specifying 
that TCAM perform a static buffer allocation (by coding PCI=(,N)). Static 
buffering causes TCAM to get all the buffers it is going to need prior to issuing 
the I/O for polling or for addressing. Processor cycles required to process the 
PCI interrupt are thus saved. However, keep in mind that if you do not use 
PCI buffering, you will need more buffers and can also experience a slower 
response time. 
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• For non-TSO or mixed TSO/non-TSO TCAM, input and output buffer sizes 
siiould be equal. That is, the output buffer size specified on the DCB, GROUP, 
or TERMINAL macro instruction should be the same as the buffer size on the 
queue. (The buffer size on the queue is determined by the input buffer size 
specified on the DCB, adjusted by special editing performed in the message 
handler.) When the buffer sizes are identical, TCAM can read the message from 
the queue in the fewest number of queue accesses and no data movement will 
occur while buffers are being built. 

Note: Buffer size on the queue does not refer to disk record lengths or 
blocksize. 

• Data is transferred between TCAM and TSO buffers on input and output in 
units of TCAM buffer unit size or TSO buffer size, whichever is smaller. To 
reduce the number of QTIP SVCs to effect the transfer of data, the TCAM 
unit size and TSO buffer size should be as large as possible. 

• On the PCB macro instruction for an application program using disk queueing, 
code BUFOUT = 2X + 1 , where X is the number of TCAM buffers needed to 
hold the largest anticipated message. The buffers allocated for BUFOUT 
constitute the application program read-ahead queue. Messages are read from 
the disk message queue into these buffers in anticipation of a GET or READ 
request from the application. If BUFOUT is too small, the result might be 
delays in satisfying application GETs or READs. 



Blocksize of Chedcpoint Data Set: The blocksize of the checkpoint data set should 
be as large as possible. The larger the blocksize, the more efficient checkpoint 
becomes; the maximum blocksize of 3520 should be used if sufficient storage is 
available. Checkpoint execution time can also be reduced by making the non- 
resident checkpoint routines resident. (Use the function-to-module list in 0S/VS2 
TCAM Logic to determine which routines to make resident.) 



TCAM Fixed Storage: TCAM performance is degraded if insufficient storage is 
available; conversely, storage is wasted if too much is specified. You can determine 
the amounts of the various storage pools (buffer units, CPBs, main-storage queue 
units, PLCBs) actually used by TCAM by examing TCAM dumps, as described in 
OS/VS TCAM User's Guide. 

In all environments, ensure the maximum utilization of the fixed storage used 
by TCAM. Since TCAM long-term fixes many of its control blocks, buffers, CPBs, 
and so on, it is important to make effective use of fixed storage. Storage is fixed 
in 4K pages; buffers, CPBs, and some control blocks cannot cross page boundaries. 
TCAM keeps track of unused allocated storage and packs control blocks into these 
areas. If the unused storage is fragmented or if more than one page is needed, 
additional pages may be allocated and fixed by TCAM unnecessarily. The number 
of fixed pages required for TCAM's control blocks can be reduced by reordering the 
opening of line groups and limiting the number of lines per line group. This and 
other techniques designed to minimize fixed pages and page faults are described in 
0S/VS2 System Programming Library: Initialization and Tuning Guide. 
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Queuing: If an MCP supports both batch and interactive processing concurrently, 
use separate queuing structures for batch processing and interactive processing 
(for example, non-reusable for batch and reusable for interactive, or vice versa). 
Otherwise, contention for the disk queue volumes might adversely affect interactive 
response time. 

When disk queuing is being used, the message queues should be separated from 
system data sets or other high volume data sets. The interaction with high volume 
data sets of any kind results in the disk not being able to handle the traffic; 
response time slows down. Using main storage queuing is one way to avoid disk 
and channel contention for the queuing medium. 

Changes in network definition can also be used to improve response time. 
When a terminal runs in batch mode part of the day and in interactive mode the 
rest of the day, you can define the same physical terminal twice to maintain the 
queuing medium separation. The interactive terminal can use one disk queue (for 
example, the reusable queue) and the batch terminal can use the other (for 
example, the non-reusable disk queue). Operator control can be used to start and 
stop the correct logical terminal. 



Consistency in Response Time: Consider use of the following features to improve 
consistency in response time : 

• The HOLD macro instruction, which suspends transmission to a particular 
terminal for a specified time and then retransmits the message that was held. 
It should be used to prevent an inoperative terminal from tying up the entire 
line. It must be used if message recovery after a permanent error is desired. 
A common error is coding too short an interval on the HOLD macro, causing 
frequent retry of a send operation. This can waste a lot of processor and line 
time if the terminal is permanently out of order. 

• For 270x or 3705 EP: 

— The MSGLIMIT macro instruction, which controls how many messages are 
sent to or received from any given terminal on a line. Use of this macro 
enhances transmission priorities. 

— The SLOWPOLL and lEDPSTOP macros, which are used to control polling 
during error conditions by preventing excessive messages to LOGREC and the 
console when a line fails. 

— Poll delay, which delays receiving for a specified time after an unproductive 
pass through the polling lists. However, for 270x or 3705 EP lines eligible for 
autopoll, do not code a poll delay. Without poll delay, polling continues on 
the line without any interruption of the processor until either a positive 
response is received or a line error occurs. With poll delay, the processor will 
be interrupted after a single pass through the polling list, and I/O must be 
restarted via EXCPVR after expiration of the interval. Note that CPRI=S 
must be coded if poll delay=0. 
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The following are general performance-related hints that should be considered 
when converting to VSAM user catalogs. For further details and additional 
suggestions on catalog conversion and VSAM catalog performance factors, see the 
Conversion Notebook and the Initialization and Tuning Guide. 

• For the conversion of large OS CVOLs, use the CNVTCAT command only to 
produce the list of catalog entries. Load the VSAM catalog via the DEFINE 
command. The CNVTCAT command will not use the full space allocated in 
your VSAM catalog since the addition of entry names in ascending order causes 
control interval spHts in the high-key range. 

• Be sure to specify a value for secondary allocation when defining the space 
requirements for VSAM catalogs. If none is specified, a "catalog full" message 
can appear when only 25% of the catalog is utilized. 

• Adjust the VSAM catalog buffer space in the DEFINE USERCATALOG and 
DEFINE MASTERCATALOG commands according to the amount of fixed 
storage available in your system. Although the storage consumption will be 
high, a large buffer space will speed up processing since more requests can be 
serviced concurrently. Also, a sufficiently large master catalog buffer space will 
maximize the number of concurrent searches of the master catalog and each job- 
shared user catalog. 

• Use one catalog for each major group of users or appHcations. Having multiple 
user catalogs shortens the time required to search for an entry and reduces 
contention for the catalogs. Thus, both processor time and wait time is saved. 
In addition, backup and recovery is made easier. However, in systems where 
storage is a critical resource, too many user catalogs can delay response time. 
Although only 200-400 bytes of real storage have to be page fixed for each 
catalog, you should consider sharing more user catalogs in storage-constrained 
systems that are experiencing unsatisfactory response time. 

• Mount the catalog volume on a non-shared device so that catalog records in 
buffers can be reused. 

• If your installation is using GDGs in VSAM catalogs, you can prevent a 
performance problem from developing by allowing extra space in the high-key 
range of the catalog. Poor utilization of the high-key range can result from the 
addition of entry names in ascending order because the space occupied by old, 
deleted generation names is not always reused as it is in CVOLs. 

If your installation has SU8 installed, however, you can create new GDGs and 
catalog them in either VSAM catalogs or OS CVOLs. Performance might be 
improved by building new index levels and GDGs in the CVOLs instead of in 
VSAM catalogs. 
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possible source of delay in satisfying I/O 
requests II.33-II.34 
DB/DC applications 

executing as non-swappable 11.78 
DB/DC environments 

(see IMS work) 
DC attribute on modules, remoying III.4 
DC Monitor 

(see IMS DC Monitor) 
DDR records 

as source of performance data 1.40 
DEBUG JES2 parameter, hint III.20 
DEFINE command of access method services 
versus CNVTCAT command for converting OS 
CVOLs III.33 
demand paging 

computing non-TCB time for VIO and demand 
paging II.15-II.19 
formula for 11.18 
effect of 2305 vs additonal storage III.l 1 
excessive, possible cause of user-oriented performance 
problem 11.63 
investigating problem II.71-II.72 
possible solutions 11.72 
focusing on 

if paging causing wait time II.5 1 -II.5 2 

target values for 11.52 
if paging costing non-productive processor time II.5 1 
if paging I/O interfering with other I/O II.49-II.51 
judging rates 11.49 
in IMS-mixed environments III.l 3-III. 14 
reducing II.55-II.58 

localizing page references II.56-II.57 
reducing fixed storage 11.56 
reducing multiprogramming level 11.58 
separating users with large working sets II.57-II.58 
steps to reduce 11.55 
target values for 11.52 
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DEQ (SVC 48) 

path length for shared DASD III.27 
path length when executed by authorized program III.27 
detected wait swaps 

possible indication of enqueue contention 11.75 
reducing 11.54 
device activity report, MF/1 or RMF 

used to compute device time per access 11.41 -11.42 
used to compute SIOs to data base 11.47 
used to identify critical I/O paths II.37-II.39 
used to identify device bottlenecks 11.41 
used to investigate possibility of I/O delays for IMS 
work 11.73 
used to monitor performance 1.45 
device bottlenecks 

(see devices, investigating) 
devices, investigating 

device measurements indicating potentially critical I/O paths 
"average time to service a request" 11.37, 11.39 
"queue length" measurement 11.37, II.38-II.39 
focusing on specific device bottlenecks II.4 1 
excessive seek time 11.41 
questionable significance of "device busy" 
measurement II.4 1 
tools that report on devices 1.29 
(see also DASD; shared DASD; tape devices; unit record 
devices) 
DFSILTAO 

(see IMS/VS transaction analysis utility program) 
DFSPIRPO 

(see IMS/VS program isolation trace report utility program) 
DFSVSAMP data set 

specifying blocksize to eliminate ISAM/OSAM data base 
compaction 11.48 
direct access contention analyzer 

(see DACA) 
direct access storage devices 

(see DASD) 
disk queueing for TCAM III. 31 
dispatcher 

processor time not recorded by . II. 10-11.11 
dispatchmg priority (DPRTY) 

assigning preferred priority to solve enqueue 

contention 11.75 
hints III. 15 

increasing, as solution to work waiting for the 
processor 11.70 
display matrix command 

as source of data 1.30 
display stations 

TGAM buffer size III.29 
distribution of system output 

operational bottleneck 1.49 
domain constraints 

(see domains weights; domains) 
domain weights 

affecting swap-in of TSO transactions 11.66 
changing MPLs vs domain weights III. 17 
importance of III. 17 
possibly causing swap-in delay 11.78 
domains 

computing TCB time for II.13-II.14 
controlling non-IMS work to reduce IMS demand 

paging III. 14 
domain control hints III. 17 
batch domains III. 17 
MPLs vs domain weights III. 17 
TSO domains III. 17 



domains (continued) 

placing TSO trivial and non-trivial transactions in separate 
domains II.66-II.67 

using to make address spaces nonswappable 11.10 
dominant users 

basic performance factor 1.47 

identifying heavy processor users II.29-II.31 

investigating paging for 11.71 
DPRTY 

(see dispatching priority) 

ended transactions field in RMF or MF/1 

deriving TSO transaction rate from 1.8 
ENQ (SVC 56) 

path length for shared DASD III.27 
path length when executed by authorized program III.27 
enqueue contention 

causing detected wait swaps 11.54 
causing enqueue exchange swaps 11.54 
causing long wait swaps 11.53 

possible cause of user-oriented performance problem 11.63 
investigation and possible solutions 11.75 
enqueue exchange swaps 

possibleindicationof enqueue contention 11.75 
reducing 11.54 
environment 

documenting when taking measurements 1.40 
error statistics 

as source of performance information I.40-I.41 
evaluation, performance 

steps 1. 1-1.2 
exchange-on-recommendation-value swaps, reducing 11.54 
exchange swaps 

(see enqueue exchange swaps; exchange-on-recommendation- 
value swaps) 
EXCPs 

increased path length in MVS III.3 

rates on MVT, as basis for judging adequacy of 

processor III.7 
reducing II.47-II.48 
benefits of 11.47 

blocksize and buffer hints III.3-III.4 
focusing on data sets with high EXCP rates III.3 
focusing on users with high EXCP rates 11.47 
for batch work 11.47 
for IMS work II.47-II.48 
for TSO work 11.47 
JES2 buffer size III.19 
possible solution to I/O delays II.33-II.34 
to paging data sets 11.47 
tools that report on 1.29 
EXCP/XDAP (SVC 0) 

example of computing TCB time for 11.23 
expectations, system 

(see performance objectives) 

FBS (fixed block standard) 

hints for DASD III.4 
FDPs 

general information 1. 1 5-1. 1 9 

input to, output from, and overhead of I.20-I.21 

operational considerations 1.2 2-1 .2 3 

types of data provided I.24-I.26 
Field Developed Programs 

(see FDPs) 
fixed block standard 

(see FBS) 
fixed-head DASD 

VTOC placement on 111.23 
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fixed storage 

maximum utilization by TCAM III.30 
reducing 

approach to reducing 11.56 
as step in reducing demand paging II.S5 
VSAM catalog buffer space III.33 
VSAM user catalogs III.33 
focus of tuning effort 

dictated by type of objective not met II.6-II.8 
forms mounting 

operational bottleneck 1.49 
formulas 

average number of pages per SIO 11.17 

average time to service an I/O request 11.39 

batch processor utilization 1.12 

channel time per access 11.41 -11.42 

device time per access II.41-II.42 

EXCP/SIO rate 11.47 

for estimating maximum MPL for TSO III.17 

for estimating processor utilization on MVS, based on MVT 

utilization III.7 
MSO service 11,57 
non-TCB time breakdown II.15-II.19 
example of using formulas 11.19 
non-swap non-paging I/O interrupt non-TCB time 11.18 
non-swap paging non-TCB time 11.18 
summary of formulas 11.16 
swapping non-TCB time 11.17 
total non-TCB time 11.15 

checking accuracy II. 16-11. 17 
percentage of processor wait time during which channel is 
busy II.37-II.38 

problem program time for a job, if SVC time is 
computed 11.31 
SVC processor cost 

as percentage of processor utilization 11.22 
as percentage of total TCB time 11.22 
SVC time for specific job II.29-II.3 1 
TCB percentage of job TCB time for specific SVC II.3 1 
TCB time from RMF 11.13 
time to resolve page-in 11.61 
TSO processor utilization 1.12 
TSO transaction rate 1.8 
working set size from RMF 11.57 
working set size from SMF II.57-II.58 
frames, number of available 

invalid as indication of real storage problems 11.49 
FREE=CLOSE JCL parameter 

possible solution to enqueue contention 11.75 
front end of SVCs 
defined 11.10 
(see also SVC usage) 

generalized trace facility 

(see GTF) 
generation data groups (GDGs) 

in VSAM catalogs III.33 
GETMAIN/FREEMAIN (SVC 10) 

example of computing TCB time for 11.23 

path length when executed by authorized program III.27 
GDGs (generation data groups) 

in VSAM catalogs III.33 
goals set for system 

(see performance objectives) 
GTF 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.22 



GTF (continued) 

reduction programs available from IBM 
general information 1. 17-1. 19 
input to, output from, and overhead of I.20-I.21 
operational considerations 1.23 
types of data provided I.25-I.26 

requesting time-stamping II.5 

types of data provided 1.24 

used to break down swapped -out time 11.78 

used to collect basic set of measurements II.5 

used to determine why swaps are occurring 11.77 

used to identify SVCs called by specific jobs/logons 11.29 

used to investigate wait-to-start time for TSO 11.66 
used to break down wait-to-start time 11.66 

used to measure nature of TCAM user activity III. 29 

used to obtain "cost" of SVCs 11.21 -11.22 

used to obtain frequency of SVCs 11.21 

used to order modules within PDS III.23 

writing data reduction program for I.41-I.42, 1.43-I.45 
GTFANAL 

general information 1.17 

input to, output from, and overhead of 1.21 

operational considerations 1.23 

types of data provided 1.26 
GTF data analysis program 

(see GTFANAL) 
GTF direct access contention analyzer 

(see DACA) 
GTF I/O concurrency tool 

(see GTFIOCUR) 
GTFIOCUR 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.23 

types of data provided 1.26 

used to report SIO condition codes 11.42 
GTF supervisor services analyzer 

(see GTFSVC) 
GTFSVC 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.23 

types of data provided 1.25 

used to identify SVCs used by a job 11.29 
meaning of "job" for GTFSVC 11.29 

used to obtain frequency of SVCs 11.21 
GTFVTAM 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.23 

types of data provided 1.26 
GTF VTAM buffer trace analysis tool 

(see GTFVTAM) 
guidelines 

(see formulas; rules-of-thumb; target values) 

happy values for SRM indicators of system contention 

changing to achieve lower multiprogramming level 11.58 
changing to reduce IMS paging III.14 
hardware 
additional 

as solution to performance problems II.8 

possibly indicated by storage shortage swaps 11.53 
conjfiguration 

basic performance factor 1.47 

(see also configuration) 
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hardware monitors 

general information 1.18 

input to, output from, and overhead of 1.21 

operational considerations 1.23 

source of data on system-wide resource utilization 11.5 

types of data provided 1.26 
HOLD macro instruction 

to improve consistency in TSO response time III. 31 
HOLD subparameter of QCONTROL parameter 

specifying for MAS configuration III.31 

identifying and reducing non-productive processor 
time II.9-II.31 

breakdown of non-TCB time II.15-II.19 

categories of processor time II.9-II.1 1 

computing TGB time n.l3-II.14 

focusing on different categories II.l 1-II.12 

identifying heavy processor users II.29-II.3 1 

SVC analysis 11.21 -n.27 
lEAFIXxx member of PARMLIB 

reducing fixed storage 11.56 
lEBCOPY 

used to re-order PDS 111.23 
IFCEREPO service aid 

as source of performance data 1.41 
IGC0004F 

isee STIMER) 
IGD0004G 

(seeTTIMER) 
IKJPRMxx member of PARMLIB 

OWAITHI value 11.53 
IMSASAP 

general information 1.19 
IMS control region 

{see control region, IMS) 
IMS DC monitor 

general information 1.18 

indicating possible I/O contention 11.73 

used in computing ratio of data-base SIOs to DB calls or 
transactions 11.48 
IMS environment 

{see IMS work) 
IMS latches 

possible cause of user-oriented performance problems 11.63 
IMS log tape analysis tool 

general information 1.19 
IMSMAP/VS data base mapping programs 

general information 1.19 
IMS measurement tools 

general information I.l 8-1.19 
IMS monitor summary and system analysis program 

(see IMSASAP) 
IMS response time 

analyzing 

(see user-oriented performance problems, analyzing) 

measurement of 1.8 
IMS transactions 

as reported on transaction response report 1.8 

factors to include in documenting workload 1. 10 
IMS utilities 

(see IMS DC Monitor; IMS/VS log transaction analysis 

utility program; IMS/VS program isolation trace report 

utility program; IMS/VS statistical analysis utility program) 
IMS work 

access to processor, investigating possible problem of 11.69 

data base segment contention, investigating possible problemi 
of n.75 



IMS work {continued) 
demand paging for 

in IMS-mixed environment III. 1 3-III. 14 
target values 11.52 

effect of pre-load 11.52 

documenting workload, factors to include in I.IO 

EXCPs for, reducing II.47-II.48 

examining ratio of data-base SIOs to DB calls or 

transactions II.47-II.48 
ways to reduce 11.48 

input queue time, investigating as possible 
delay II.65-II.66 

not executed in APG in.l5 

real storage guidelines III.9 

reducing fixed CSA for 11.56 

shared DASD, possible impact of III.25 

significant paging factor for 11.71 

waiting for I/O, investigating as possible delay 11.73 
IMS/VS DC Monitor 

{see IMS DC Monitor) 
IMS/VS log transaction analysis utility program 

general information 1.18 

used to investigate input queue time 11.65 

used to investigate possible problem of work waiting for the 
processor 11.69 
IMS/VS program isolation trace report utility program 

general information 1,18 

used to investigate possible data base contention 11,75 
IMS/VS statistical analysis utility programs 

general information 1,18 

measurements of IMS response time 1,8 

used to compute ratio of data-base SIOs to DB calls or 
transactions 11.47 

used to investigate possible problem of work waiting for 
the processor 11.69 
IMS/VS virtual storage analysis tool 

general information 1.19 
"index-in-core" option for ISAM 

reducing IMS SIOs 11.48 
initialization parameters 

addressing as basic performance factor 1.47 

JES2, hints in.l9-III.21 
initiators, number of 

as possible solution to excessive wait-to-start time 11,65 

controlluig non-IMS work to reduce IMS demand 
paging III.14 

operational bottleneck 1.48 
input buffers 

size for TCAM III.30 
input of work into system 

operational bottleneck 1.49 
input queue time, excessive 

(see waiting to start) 
input terminal wait swaps, reducing 11.53 
Installed User Programs 

(see lUPs) 
introduction to performance evaluation I.1-I.2 
investigating the use of I/O reso urces 

(see I/O resources, investigating) 
investigating the use of processor time 

(see processor time, investigating) 
investigating the use of real storage 

(see real storage, investigating) 
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I/O activity 

measured by SMF 1,43 
measuring via GTF 1.44 

tr acing, as part of basic set of measurements II .5 
(.see also I/O resources) 
I/O bottlenecks 

(see I/O resources, investigating) 
I/O device activity report 

(see device activity report, MF/1 or RMF) 
I/O interrupts 

approximate non-TCB time for processing 11.15 
non-swap, non-paging 

computing processor time for II.15-II.19 
formula for 11.18 
non-swap paging 

computing processor time for 11,15-11.19 
formula for 11.18 
I/O requests 

steps in satisfying and their performance 
implications 11,33-11.34 
I/O resources 

as factor in documenting workload 1,10 
balancing 

basic performance factor 1.47 
configuration 

guidelines for MVS III.8 
optional channels III.5 
control blocks that contain information on 1.29 
I/O resources, investigating II. 33-11.48 

delays in satisfying requests, overview 11.33-11,34 
focusing on specific I/O bottlenecks 11.41-11,43 
identifying critical I/O paths 11.37-11,39 
paging 1/ O interfering with other I/O II.49-II.5 1 
reducing bottlenecks in I/O paths 11.45 
reducing EXCPs/SIOs 11.47 
summary of process to investigate 11,35 
when to investigate 11,33 
I/O, waiting for 

possible cause of user-oriented performance problems 11,63 
investigation and possible solutions 
IMS 11,73 

TSO or batch work 11,73 
I/O trace records 

providing information on I/O activity 1.44 
IPS performance objective 

affecting unilateral and exchange-on-recommendation- 
value swaps 11.54 
assigning preferred objectives to work to solve enqueue 

contention 11.75 
distinction from system expectations 1,5 
possibly affecting swap-in-delay 
after swap-out 11.78 
waiting to start for TSO 11.66-11,67 
specifjdng to ensure pre-emption 11.67 
IPS specification 

(see IPS performance objective) 
ISAM data bases 

ways to reduce data base SIOs 11,48 
ISAM "index-in-core" option 
reducing IMS SIOs 11,48 
iterative nature of performance analysis II. 3 
lUPs 

general information I.15-I.19 
input to, output from, and overhead of I.20-I.21 
operational considerations 1,22-1.23 
types of data provided 1.24-1,26 
IWAIT time, high 

indicating possible I/O contention 11.73 



JCL parameter FREE=CLOSE 

possible solution to enqueue contention 11.75 
JESGEN 

JES2 generation and initialization parameter 
hints 111.19-111,21 
JESGEN listing 

as source of data 1.30 
JES2 

buffer size hints III. 19 

job journaling and SMF records III.20 

not run in APG III.15 

QCONTROL in MAS configuration III.21 

real storage guidelines III.9 

Sn initialization parameter III.21 

&DEBUG hint III.20 

&NUMDA hints III.20 

&NUMTGV hints III.19-III.20 
JES2 generation 

(see JESGEN) 
JES2 initialization 

(see JESGEN) 
JES3 

real storage guidelines III. 9 

SDEPTH parameter 

affecting volume mounting 1.20 
JES3 monitoring facility 

(see JMF) 
JIA 

general information 1.17 

input to, output from, and overhead of 1,20 

operational considerations 1.23 

types of data provided 1,25 

comparison to SUDP and SIR 1,27 
JMF 

general information 1,18 
job class initialization parameters 

subparameters to specify to reduce overhead III.20 
job classes 

factors to include in documenting workload 1.10 

installation controls on 1.12 

restructuring 

as possible solution to excessive wait-to-start time 11.65 

sample objective for 1.13 
job entry subsystem 

(see JES2;JES3) 
job journaling hint III. 20 
jobstream tuning 

using SMF data for 1,42 
job termination SMF record 

(see type 5 SMF record) 
journaling hint, job 111,20 

latch contention 

possible cause of user-oriented performance problems 11.63 
link pack area 

(see LP A) 
local system queue area 

(see LSQA) 
lock contention 

possible cause of user-oriented performance problems 11,63 
logical channel paths 

balancing activity within physical path III.5 
LOGREC data set 

preventing excessive messages when line fails III. 31 

source of performance information 1,40-1.4 1 
log transaction analysis utility program 

(see IMS/VS log transaction analysis utility program) 
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long wait swaps 

possible indication of enqueue contention 11.75 

reducing 11.53 
LPA (link pack area) 

fixed, reducing 11.56 

real storage guidelines III.9 
LSQA (local system queue area) 

fixed frames for 11.56 

real storage guidelines III.9 

main storage service 
(see MSO service) 
MAS configuration 

specifying QCONTROL III.21 
master catalog 

(see VSAM catalogs) 
master scheduler 

not run in APG III. 15 
real storage guidelines III.9 
MAXDORM parameter of QCONTROL parameter 

specifying for MAS configuration III.21 
maximum CPU service units per second 11.14 
maximum MPL 

affecting swap-in of TSO transactions 11.66 
for batch III.17 
for TSO domains III.17 
MAXJOBS 

possible effect of &NUMTGV on III.19 
specifying III. 21 
MCH records 

as source of performance data 1.40 
mean-time-to-wait subgroup of APG 

(see MTTW subgroup of APG) 
measurement tools 

executing on regular basis 

for monitoring performance 1,45-1.46 
to identify usual values of measurements II.5 
importance of 1.15 
monitoring performance 1. 45-1.46 
other sources of MVS data I.30-I.41 
control blocks 1.30-1.34 
SYS1.L0GREC I.40-I.41 
trace table 1.35 -1.40 
overview of IBM MVS measurement tools I.15-I.30 
comparison of SIR/SUDP/JIA 1.27 
general information I.17-I.19 
input to, output from, and overhead of I.20-I.21 
operational considerations 1.2 2-1.23 
sources of data on system resources I.28-I.29 
types of data provided I.24-I.26 
writing data reduction programs 1.4 1-1.45 
GTF data 1.4 1-1.42, 1.4 3-1.45 
SMFdata I.41-I.43 
trace table 1. 35 -1.40 
measurements 

basic set to begin analysis II.5-II.6 

judging values of specific measurements II.5 
types of information to collect II.5 
on address-space level 

needed to analyze user-oriented performance 
problems 11.63 



message processing regions (MPRs) 

executing as nonswappable 11.78 

restructuring 

to solve excessive input queue time 11.65 
message queue buffers, IMS 

insufficient 11.66 

MF/1 

executing with SMF 1.42 

general information 1.17 

input to, output from, and overhead of 1.20 

measurement of TSO response time 1.7 -1.8 

measurement of TSO transaction rate 1.8 

measurement of wait time II.7 

operational considerations 1.22 

source of information on system-wide resource 
utilization II.5 

types of data provided 1.24 

used to compute SIOs to data base 11.47 

used to focus on specific I/O bottlenecks 
device bottlenecks 11.41 
high chaimel time per access 11.41 -11.42 

used to identify critical I/O paths II.37-II.39 

used to investigate demand paging 11.57 

used to investigate possibility of I/O delays for IMS 
work 11.73 
MF/1 CPU activity report 

(see CPU activity report) 
MF/1 post analyzer 

(see POSTANAL) 
MF/1 workload activity report 

(see workload activity report, MF/1) 
MINDORM subparameter of QCONTROL parameter 

specifying for MAS configuration III.21 
minimum MPL 

(see MPL) 
modifying system control program 

as solution to performance problem II.8 
module packing 

to achieve efficient page reference patterns 11.57 
module reference patterns 

possible solution to excessive paging 11.72 
monitoring performance I.45-I.46 

as step in performance evaluation 1.2 
monitor summary and system analysis program 

(see IMSASAP) 
monitors 

(see hardware monitors; measurement tools) 
monopolizers of system resources 

(see dominant users) 
MOUNT processing 

causing long wait swaps 11.53 
mounting, volume 

(see volume mounting) 
MP 

processor guideline for converting from two UPs IIL8 
MPL 

affecting swap-in of TSO transactions 11.66 

changing MPLs via domain weights III. 17 

for batch III.17 
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MPL (continued) 

forTSO III. 17 

importance of I II. 1 7 

lowering to achieve reduced demand paging 11.58 

setting to control non-IMS work III. 14 

specifying to make address spaces nonswappable 11,10 
MPRs 

(see message processing regions) 
MSGLIMIT macro instruction 

to improve consistency in TSO response time III.3 1 
MSO service 

formula used by SRM 11.57 
MTTW subgroup of APG 

for TSO work III.15 

placement of work above MTTW to increase access to 
processor 11,70 
multi-access spool (MAS) configuration 

specifying QCONTROL III.21 
multiprocessors 

(see MP) 
multiprogramming level, reducing 

as step in reducing demand paging 11.55 

by lowering MPL for specific domains 1 1.5 8 

by resetting threshold values 11,58 

(see also MPL) 
MVS measurement tools 

(see measurement tools) 
MVS seek analysis program 

(see SEEKANAL) 
MVS storage utilization display 

(see SUDP) 

NCP 

hints for DASD 111,3-111.4 
network definition 

affecting TSO response time III.3 1 
NOJOURN 

reducing overhead III. 20 
NOLOG 

reducing overhead III.20 
non-reusable disk queueing 

for batch processing supported by TCAM MCP III .3 1 
non-swap, non-VIO paging 

(see demand paging) 
non-swap paging 

computing non-TCB time for II. 1 5 -II. 1 9 
formula for 11.18 
nonswappable work 

effect of privileged bit in PPT on TCB time reported by 

RMF 11.10 
recommendations for 11.78 
resulting in requested swaps 11.53 
using domains to achieve 11.10 
non-TCB time 

approximate times for SCP functions 

for ASM processing for non-swap paging 11.15 
for ASM processing for swap paging 11.15 
for I/O interrupt processing 11. 1 5 
for RCT processing 11.15 
breakdown of non-TCB time II.15-II.19 
example of using formulas 11.19 
formula for non-swap non-pagjng I/O interrupt non-TCB 
time 11.18 

formula for non-swap paging non-TCB time II. i 8 
formula for non-TCB time 11.15 

testing formula 11.16 
formula for swapping non-TCB time II. 1 7 
summary of formulas II. 1 6 
NOTE/POINT, hint III.3 



NOTYPE26 

reducing overhead III. 20 
NOUJP 

reducing overhead III. 20 
NOUSO 

reducing overhead 111,20 
nucleus 

real storage guidelines III.9 

reducing to reduce fixed storage 11.56 
NUMDA JES2 parameter, hint 111.20 
NUMTGV parameter 

(see &NUMTGV) 

objectives 

(see performance objectives; IPS performance objectives) 
OBR records 

as source of performance data 1.41 
operating procedures 

(see operational bottlenecks) 
operational bottlenecks 
avoiding 1.48-1,49 

difficulty of recognizing and measuring 1.48 
forms mounting 1,49 
input and output 1,49 
number of initiators 1.48 
reply delays 1,49 
shift changes 1,49 
volume mounting 1,49 
basic performance factor 1.47 , II.6 
impacting I/O requests 1,33 
shared DASD 111,25 
OPTCD=C 

(see chained scheduling) 
OPTCHAN 

(see optional channel path) 
optional channel path 

balancing activity with primary path 111,5 
string switching's effect on 111,25 
to minimize effect of hardware failure on shared 
DASD in,26 
OSAM data bases 

ways to reduce data base SIOs 11.48 
OS CVOLs 

(see CVOLs) 
OS/VS capacity management aid 

(see CMA) 
OUCB 

as source of performance data 1,33-1,34 
output buffers 

insufficient, causing output terminal wait swaps 11.53 
size for TCAM 111,30 
output of work from system 

operational bottleneck 1,49 
output terminal wait swaps, reducing 11,53 
OUXB 

as source of performance data 1,34 
overhead 

minimizing when writing SMF and GTF data reduction 

programs 1.41 
of measurement tools 1.20-1,21 
reducing 

job journaling 111,20 

number of Sn initialization parameters 111,21 
unused SMF exits 111,20 
OWAITHI value in IKJPRMxx 

affecting output terminal wait swaps 11,53 

packlists 

using GTF data to create 1.44 
pageable storage, increasing size of 

(see fixed storage, reducing) 
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page data set activity report, RMF 

using to monitor performance 1.46 
page data sets 

examining to reduce time to resolve page-in 11.61 
placement 

affecting swap-in time 11.66 
reducing EXCPs to 11.47 
page fault 

(see paging) 
page fault rate, SRM indicator of system contention 
changing threshold value to reduce multiprogramming 
level 11.58 
page frame table (PFT) 

as source of performance data 1.32 
page-in, time to resolve, computing 11.61 
page reclaims 

{see reclaims) 
page reference patterns, achieving efficient 
as step in reducing demand paging II .55 
investigating II.56-II.57 
"page seconds" in SMF records 4/34 

used to compute working set size II.57-II.58 
page-stealing algorithm in SU7 

possible effect on IMS paging III. 13 
page vector table (PVT) 

as source of performance data 1.32 
paging 

demand paging, reducing 11.55 -11.58 

target values for demand paging 11.52 
effect of 2305 vs. additional storage III. 11 
examining to identify real storage problems II.49-II.52 
paging causing wait time II.5 1 -II.5 2 
paging costing non-productive processor time 11.51 
paging I/O interfering with other I/O II.49-II.5 1 
minimizing in TCAM fixed storage 111,30 
non-swap paging 

computing non-TCB time for II.15-II .19 
formula for 11.18 
reclaims 

investigating for batch applications 1 1.7 1 
investigating on system level 11.49 
swap paging 

computing non-TCB time for II. 1 5 -II. 1 9 

formula for 11.17 
reducing II.53-II.54 
time to resolve page faults, reducing 11.61 
VIO paging, reducing 11.59 
paging activity report, RMF 

used to compute average time to resolve page-in 11.61 
used to compute non-TCB time TI. 15-11. 16 
non-swap paging rate 11.15 
swap paging rate 11.15 
swap rate 11.15 
used to draw storage map 11.55 
used to examine paging rates II.49-II.51 

sample report 11.50 
used to investigate possible enqueue contention on system 
level 11.75 
used to monitor performance 1.46 
used to report swaps 11.53 
paging data sets 

(see page data sets) 



PARMLIB 

OWAITHI value in IKJPRMxx II.5 3 
lEAFIXxx, reducing fixed storage 11.56 
listing, as source of data 1.30 
partitioned data sets (PDSs) 

module ordering within III.23 
path lengths 

reducing for SVCs 
ENQ/DEQ III.27 

executed by authorized programs III.27 
STIMER and TTIMER III.27 
(see also instruction counts) 
PCF (programming control facility) 

obtaining TCB time for commands 11.29 
PCI processing, reducing 

for program libraries III.4 
TCAM buffer size III.29 
PDS 

module ordering within III.23 
"percent channel busy" measurement 

as indication of possibly critical I/O path 11.37, 11.38 
"percent channel busy and CPU wait" measurement 
as indication of possibly critical I/O path II.37-II.38 
combining with "CPU wait" measurement 11.38 
performance evaluation 

steps L1-I.2 
performance problem analysis 

(see analysis, performance problem) 
performance groups 

computing TCB time for II. 1 3-II. 14 
performance objectives 1.5-1,12 

as basis of performance problem descriptions II.4 
distinction from IPS performance objectives 1.5 
documenting the workload 1,9-1, 1 
measuring resource requirements 1, 1 1 -1. 1 2 
modifying, as solution to performance problems II.8 
purpose 1 .5 
sample objective 1.13 
selecting and measuring objectives I.6-I.9 
setting objectives 1,11 
steps in defining 1,6 

(for IPS performance objectives, see IPS performance 
objective; see also system-oriented performance 
objectives; user-oriented performance objectives) 
PFT (page frame table) 

as source of performance data 1.32 
physical channel path 

balancing activity on logical channels III.5 
planning for a performance evaluation 

defining performance objectives 1,5-1,13 
pre-initialization MVS performance factors I.47-I.49 
selecting measurement tools 1 ,15-1,46 
PLPA 

demand paging in, reported by SIR 11.57 
examining placement to reduce time to resolve 
page-in 11.61 
POST (SVC 2) 

reduced path length when executed by authorized 
program III.27 
POSTANAL 

general information 1.17 
input to and output from 1.20 
operational considerations 1.23 
types of data provided 1.25 
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PPT (program properties table) 

effect of privileged bit on service units reported by 
RMF 11.10 
pre-emption of non-trivial transactions by trivial transactions 
ensuring, to prevent waiting-to-start delays 11.67 
preferred solution to waiting-to-start delays 11.67 
warning 11.67 
pre-initialization MVS performance factors I.47-I.49 
dominant users 1.47 
hardware configuration, adequacy of 1.47 
I/O resource balancing 1.47 
operational bottlenecks I.47-I.49 
sysgen and initialization parameters 1.47 
pre-load in IMS 

effect on demand paging rates 11.52 
preparing for a performance evaluation 

(see planning for a performance evaluation) 
primary channel path 

balancing activity with optional path III.5 
priority of work 

factor in defining categories of work 1.9 
scheduling priority of IMS transactions 

as cause of excessive input queue time 11.65 
(see also dispatching priority) 
private area of real storage 

reducing fixed storage in 11.56 
privileged address spaces 

effect on service units reported by RMF 11.10 
problem analysis 

(see analysis, performance problem) 
problem description 

as basis of performance analysis II.4 
problem program time 

computing, if SVC time has been computed 11.31 
in relation to other categories of processor time 11.10 
procedure libraries 

blocksize hints III.4 
procedure library listings 

as source of data 1.30 
processor 

coefficients reflecting relative time to perform 
SCP functions on different models 11.17 
used in converting SVC time measured on one processor 

to approximate time on different processor 11.22 
used in formulas for breakdown of non-TCB time II. 15-11. 19 
rules-of-thumb for judging reasonableness of III.7-III.8 
stopping, using QUIESCE command II. 1 1 
waiting for 

possible cause of user-oriented problems 11.63 
investigating 11.69 
possible solutions 11.70 
processor service units 

computing TCB time from 11.13 
maximum per second 11.14 
not recorded by RMF 11.10 
processor time, investigating 

breakdown of non-TCB time II. 1 5-II. 1 9 
categories of processor time II.9-II.1 1 
computing TCB time II.l 3-II.14 
due to paging, nonproductive II.5 1 
focusing on different categories II.l 1-II.12 
hints for reducing 

blocksize and buffer hints III.3-III.4 
multiple user catalogs III. 33 
identifying and reducing non-productive processor 

time II.9-II.31 
identifying heavy processor users II.29-II.31 

summary of steps 11.30 
SVC analysis II.21-II.27 

computing processor time of SVCs II.21-II.22 
ways to reduce criticalness of processor II.9 
when to investigate II.9 



processor, measurements of 

control blocks that contain information on 1.28 

factor in documenting workload 1. 10 

from GTF 1.44 

from SMF 1.42 

tools that report on 1.28 
processor utilization 

formula to compute for batch work 1.12 

formula to compute for TSO work 1.12 

in MVS, compared to MVT or SVS III.7-III.8 

on MP, compared to two UPs III. 8 
processor wait time 

causes of II.7 

due to paging, investigating 11.51-11,52 

due to I/O 11.33 

(see also I/O resources, investigating) 

during which channel is busy, computing 11.38 

presence of, dictating direction of analysis II.7 

(see also processor time, investigating) 
program-controlled interrupt 

(see PCI processing, reducing) 
program isolation feature of IMS 

effect on solutions to data base segment contention 11.75 
program isolation trace report utility program 

(see IMS/VS program isolation trace report utility program) 
program libraries 

blocksize hints III.4 
programming control facility 

(see PCF) 
program properties table 

(see PPT) 
programs 

identifying those with high TCB time 11.29 

lack of SVC information for 11.29 
PRTOV 

and OPTCD=C III. 3 
PRTY parameter of TRANSACT macro instruction 

possible solution to excessive input queue time 11.65 
PVT (page vector table) 

as source of performance data 1.32 

QCONTROL JES2 initialization parameter 

in MAS configuration III.21 
QSAM 

to DASD III.3 

to tape III. 3 

to unit record devices III. 3 
queue length measurement 

as indication of possibly critical I/O path 11.37, II.38-II.39 

used to compute average time to service a 
request 11.37, 11.39 
queueing for TCAM III.3 1 
queue time, input 

(see waiting to start) 
queued sequential access method 

(see QSAM) 
QUIESCE operator command 

using to stop processor II.l 1 

RCT processing 

approximate non-TCB processor time for 11.15 
real storage 

additional storage vs. 2305 III.l 1 

rules-of-thumb for estimating MVS require- 
ments III.8-III.10 
real storage, investigating 

demand paging, reducing II.55-II.58 

examining paging rates as indication of problem 11.49 
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real storage, investigating {continued) 

hints related to real storage 

blocksizes/buffers, trade-off III. 3 

on DASD III.4 
JES2 buffer size III.19 
TCAM fixed storage 
buffer hints III.29 
ensuring maximum utilization III. 30 
VSAM catalog buffer space III.33 
VSAM user catalogs III. 3 3 

situations triggering real storage investigation II.49-II.52 
paging causing wait time II.51-1I.52 
paging costing non-productive processor time 11.51 
paging I/O interfering with other I/O II.49-II.5 1 

swapping, reducing II.53-II.54 

time to resolve page faults, reducing 11.61 

VIO paging, reducing 11.59 
real storage, measurements of 

control blocks that report on I.28-I.29, 1.30 

factor in documenting workload 1. 10 

formula used by SRM to compute MSO service units 11,57 

measured by SMF 1.4 2-1.4 3 

tools that report on I.28-I.29 
real storage shortage swap, reducing 11.53 
reasonableness of objectives, judging 1. 1 2 
RECFM 

hints for DASD III.4 

hints for tape III. 3 
reclaims 

investigating for batch applications II.7 1 

investigating on system level 11.49 
reconnects from RPS devices 

possible channel bottleneck 11.42 
record format 

(see RECFM) 
record type 4, SMF 

(see type 4 SMF record) 
record type 5, SMF 

(see type 5 SMF record) 
record type 15 , SMF 

(see type 15 SMF record) 
record type 34, SMF 

(see type 34 SMF record) 
record type 35 , SMF 

(see type 35 SMF record) 
records type 70-76, SMF 

(see type 70-76 SMF records) 
reduction programs 

(see data reduction programs) 
region control task processing 

(see RCT processing) 
reply delays 

operational bottlenecks 1.49 
requested swaps, reducing 11.53 
RESERVE interference 

investigating as possible device bottleneck 11,41 
residency time 

computing from RMF for TSO trivial transactions 11.66 

used in investigating if work has access to processor 11,69 

used in investigating possibility of excessive swapping 11.77 
resource limits, setting 1.12 
resource management 

focus of analysis for system-oriented problems II.6-II.7 

focusing on when workload management trade-offs cannot 
be made II.8 
resource measurement facility 

(see RMF) 



resource requirements of work 

factor in documenting workload 1.9-1.10 

measuring 1. 11-1.12 
response-oriented systems 

(see CICS environment; IMS work; TSO work) 
response time 

(see IMS response time; TSO response time) 
reusable disk queueing 

for TCAM III.31 
RMCT (SRM control table) 

as source of performance data 1.32 
RMF 

executing with SMF 1,42 

general information 1, 1 7 

input to, output from, and overhead of 1,20 

measurement of TSO response time 1.7 -1.8 

measurement of TSO transaction rate 1.8 

measurement of wait time II. 7 

operational considerations 1.22 

source of information on system-wide resource utilization II.5 

TCB processor service units not recorded by 11.13 

types of data provided 1.24 

used to compute average time to resolve page-in 11.61 

used to compute EXCP/SIO rates 11.47 

used to compute residency time for TSO trivial 
transactions 11.66 

used to compute SIOs to data base 11.47 

used to compute TCB time II. 13 -II. 14 

used to compute wait-to-start time for TSO trivial 
transactions 11,66 

used to compute working set size 11.57 

used to draw storage map 11.55 

used to examine demand paging 11.57 

used to examine paging II.49-II.51 

used to focus on specific I/O bottlenecks 
device bottlenecks 11.41 
high channel time per access II.41-II.42 

used to identify critical 1/0 paths II.37-II.39 

used to investigate possible enqueue contention on system 
level 11.75 

used to investigate possibility of excessive swapping 11.77 

used to investigate possibility of I/O delays for IMS 
work 11.73 
RMF CPU activity report 

(see CPU activity report, MF/1 or RMF) 
RMF paging activity report 

(see paging activity report, RMF) 
RMF trace 

used to compute average time to resolve page-in 11.61 
RMF workload activity report 

(see workload activity report, RMF) 
rotate priority 

used to increase work's access to processor 11.70 
rotational delay 

possible source of delay in satisfying DASD 1/0 
requests II. 33-11.34 
RPS DASD 

affect of channel scheduling on III.5 

channel contention caused by reconnects 11.42 

channels to 

judging reasonableness of III.8 

failure to use SET SECTOR causing possible channel 
bottleneck 11.42 
rules-of-thumb 

configuration III.7-III.10 

for judging demand paging by specific batch 
applications 11.72 
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rules-of-thumb (continued) 

for judging excessive swapping for batch work 11.77 
forjudging if work is being delayed before being 

swapped back in 11.77 -II. 7 8 
{see also target values) 

SAM 

{see sequential access method) 
scheduling priority of IMS transactions 

as cause of excessive input queue time 11.65 
SCP functions 
{see system control program functions) 
SDEPTH parameter of JESS 

affecting volume mounting 1.49 
secondary allocation for VSAM catalogs III.33 
SEEKANAL 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.23 

types of data provided 1.25 

used to identify excessive SEEK time 11.41 

used to order modules within PDS III.23 

used to report SIO condition codes 11.42 
seek analysis program 

{see SEEKANAL) 
seek time 

investigating as possible device bottleneck 11.41 

possible source of delay in satisfying I/O 
requests 11.33-11,34 

reducing 

data set placement III.23 
module order within PDS III.23 
&NUMTGV specification III.19-III.20 
selecting measurement tools 

{see measurement tools) 
selector channels 

balancing optional channel activity on III.5 

judging utilization of 11.38 

tapes and 3270s on III.l 1 
sequential access method 

blocksize and buffer hints III.3-III.4 
service units 

{see MSO service units; processor service units) 
SET SECTOR for RPS devices 

failure to use causing possible channel bottleneck 11.42 
shared DASD systems 

cc=l from SIO, invalid as indication of control unit 
contention 11.43 

ENQ/DEQ path length III.27 

library maintenance III.26 

operational problems III.25 

possible performance degradation III.25 

RESERVE interference, investigating 11.41 

string switching and isolating shared DASD III.25-III.26 

VSAM catalog on non-shared device III.33 
shared job queue (MAS configuration) 

specifying QCONTROL parameter III.21 
shift changes 

operational bottleneck 1.49 
SIO condition codes 

code 1, indicating possible control unit 
bottleneck II.42-II.43 
inapplicable to shared DASD 11.43 

code 2, indicating possible channel bottleneck 11.42 

tools that report 11.42 
SIO trace records 

providing information on I/O activity 1.44 
SIOs, reducing 

{see EXCPs, reducing) 



SIR 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.22 

providing demand paging information 11.57 

providing working set sizes 11.57 

source of data on resource utilization by address 

space 11,38 
TCB and SRB time not reported by II.lO-II.l 1 
types of data provided 1.25 

used to examine fixed storage in private areas 11.56 
used to investigate enqueue contention 11.75 
used to investigate excessive swapping on address-space 

level 11.77 
used to investigate paging on address space/job level 11.71 

providing information on where page faults occur 11.72 
used to investigate possible problem of work waiting for 

the processor 11.69 
used to monitor performance 1.46 
used to report on dominant users 1.47 
SIRBATCH 
{see SIR) 
SIRTSO 

{see SIR) 
SLOWPOLL macro instruction 

to improve consistency in TSO response time III.31 
SMF 

analyzing for data sets with high EXCP counts III.3 
analyzing for excessive wait-to-start time for batch 11.65 
breakdown of demand paging by address space 11.57 
data on batch turnaround and throughput 1.9 
examples of information to obtain from 1.42-1,43 
executing with RMF or MF/1 1.42 
general information 1.17 
input to, output from, and overhead of 1.20 
operational considerations 1.22 
reduction programs available from IBM 

{see CMA; POSTANAL) 
reduction programs, source of data on resource utilization 

by address space II.5 
reduction programs, writing I.41-L43 
source of reclaim rates 11.57 
TCB and SRB time not reported by 11.10 
types of data provided 1.24 
used to compute working set size II.57-II.58 
used to focus on heavy processor users 11.29 
used to investigate paging on job level 11.71 
used to investigate possibility of excessive swapping 
work swapped too frequently 11.77 
work delayed before being swapped back in II.77-II.78 
used to investigate possible problem of waiting for I/O 

(TSO or batch work) 11.73 
used to investigate possible problem of work waiting for 
the processor 11.69 
used to investigate VIO paging 11.59 
used to obtain EXCP counts 11.47 
used to report on dominant users 1.47 
SMF exits 

enforcing resource limits I.l 2 
turning off to reduce overhead III.20 
Sn JES2 initialization parameter 

number to specify to reduce overhead III.21 
SPCT 

as source of performance data 1.3 3 
spool volumes, number of (JES2) 111,20 
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SQA (system queue area) 

difficulty of reducing fixed storage in 11.56 

real storage guidelines IIL9 
SRB time 

as possible indicator of I/O activity 1 .4 3 

from SMF 1.42 

in relation to other categories of processor time 11.10 

not focused on in analysis 11.12 

not reported by SIR 11.10 

not reported by SMF 11.10 

types of services that result in SRB time 11.12 
SRM 

changes in SU7, effect on IMS demand paging III.13-III.14 

definition of TSO transaction I.7-I.8 

domain control hints III. 17 

formula used to compute MSO service 11.57 

(see also AFG; domain weights; domains; happy values; 
IPS performance objective; MPL; MTTW subgroup of APG) 
SRM control table (RMCT) 

as source of performance data 1.32 
SRM events 

tracing as part of basic set of measurements II.5 
SRM user control block (OUCB) 

as source of performance data 1.3 3-1.34 
SRM user extension block (OUXB) 

as source of performance data 1.34 
SRM trace records from GTF 

used to break down swapped -out time 11.78 

used to determine why swaps are occurring 11.77 

used to investigate wait-to-start time for TSO 11.66 
used to break down wait-to-start time 11.66 

used to monitor SYSEVENT activity 1.45 
started tasks 

computing TCB time via RMF II.13-II.14 

not reported by SMF 11.10 
static buffering for TCAM III.29 
statistical analysis utility programs 

(see IMS/VS statistical analysis utility programs) 
STC 

(see &STC) 
STIMER 

causing long wait swaps 11.53 

reducing path length III.27 
stopping the processor 

using QUIESCE command II. 1 1 
storage 

(see real storage; virtual storage) 
storage map 

drawing to visualize storage usage 11.55 
storage shortage swaps, reducing II.5 3 
storage utilization display 

(see SUDP) 
string switching 

for shared DASD III.25-III.26 
SUDP 

general information 1.17 

input to, output from, and overhead of 1.20 

operational considerations 1.22 

types of data provided 1.24 

comparison to SIR and JIA 1.27 
supervisor services 

tracing, as part of basic set of measurements II.5 

using GTF to measure processor time for 1.44 
supervisor state 

in MVS, compared to MVT III.7 
supervisor time 

in relation to other categories of processor time 11.10 



SU5 

(Note: Information in this book is based on a 3.7 MVS 
system with SU 5 and SU 7 applied; no specific information 
on SU 5 is included.) 
SU7 

(Note: Information in this book is based on a 3.7 MVS 
system with SU 5 and SU 7 applied; following index entries 
refer to specific information on SU 7.) 
SRM changes, effect on IMS paging III. 1 3-III. 14 
SUB 

using GDGs in VSAM catalogs III.33 
SVC trace records 

measuring processor time spent on supervisor services 1.44 
SVC time 

in relation to other categories of processor time 11.10 
SVC usage 

analyzing II.21-II.27 

computing pro cessor time of SVCs II.2 1 -II.2 3 
example of TCB times for SVCs II.24-II.27 
frequency of SVCs 11.21 
authorizing programs to reduce SVC path lengths III.27 
ENQ/DEQ for shared DASD III.27 
focusing on during analysis 11.11 
focusing on SVC time when identifying heavy processor 
users II.29-II.31 
computing SVC time for a job II.29-II.3 1 
computing TCB time used for each SVC by a job II.3 1 
STIMER, reducing path length III.27 
TCB time 

"front end" defined 11.10 

in relation to other categories of processor time 11.10 
TPUT with HOLD 

causing output terminal wait swaps 11.53 
tracing as part of basic set of measurements II. 5 
TTIMER, reducing path length III. 27 
using GTF to measure processor time for 1.44 
SVCO 

(see EXCP/XDAP) 
SVCl 

(see WAIT) 
SVC 2 

(see POST) 
SVC 10 

(see GETMAIN/FREEMAIN) 
SVC 46 

(see TTIMER) 
SVC 47 

(see STIMER) 
SVC 48 

(see DEQ) 
SVC 56 

(see ENQ) 
SVS/MVS system and job impact analyzer 

(see JIA) 
swap communication table (SPCT) 

as source of performance data 1.33 
swap data set activity report, RMF 

using to monitor performance 1.46 
swap-in delay after swap out 
investigating II.77-II.78 
possible solutions 11.78 
swap-in delay for TSO users (waiting to start) 
investigating II.66-II.67 
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swap paging 
focusing on 

if paging costing non-productive processor time 11.51 
if paging I/O interfering with other I/O II.49-II.5 1 
judging rates 11.49 
reducing II.53-II.54 
swapping 

computing non-TCB time for II. 15 -II. 19 

formula for 11.17 
effect of 2305 vs additional storage III.l 1 
excessive, possible cause of user-oriented performance 
problems 11.63 
investigating problem II.77-II.78 
work delayed before being swapped 

back in 11.77 -11.78 
work swapped too frequently 11.77 
possible solutions 11.78 
reducing II.53-II.54 

using SMF data to obtain rates, warning 1.43 
SYSEVENT activity 

monitoring via GTF 1.45 
sysgen parameters, addressing 1.47 
SYSGEN stage 1 listing 

as source of data 1.30 
SYSRES 

VTOC placement on III.23 
system activity measurement facility 

(seeMF/1) 
system and job impact analyzer 

(see JIA) 
system control program (SCP) functions 

coefficients reflecting relative time to perform SCP 
functions on different processor models 11.17 
used in converting SVC time measured on one processor 

to approximate time on different processor 11.22 
used in formulas for breakdown of non-TCB time II. 15-11.19 
system expectations 

(see performance objectives) 
system generation 
(see SYSGEN) 
system information routines 

(see SIR) 
system management facilities 

(see SMF) 
system-oriented performance objectives 
as basis of problem descriptions II.4 
batch throughput 1.9 
definition 1.6 

dictating focus of analysis II.6-II.7 
possible conflict with user-oriented objectives 1.6 
relationship to resource management II.6 
TSO transaction rate 1.8 
system-oriented performance problems, analyzing 
focusing on major resources II.7 
(see also I/O resources, investigating; processor time, 
investigating; real storage, investigating) 
system queue area 

(see SQA) 
system resources manager 

(see SRM) 
system wait time 

(see processor, wait time) 
SYSl.LOGREC 

source of performance information 1. 4 0-1.4 1 
SYS1.PARMLIB 
(see PARMLIB) 

tape devices 

and 3270s, configuration of III.l 1 

buffer pool considerations III.3 

tools that report on 1.29 
target MPL 

affecting swap-in of TSO transactions 11.66 



target values 

demand paging 11.52 

forjudging excessive swapping for TSO work 11.77 

precent-channel-busy measurement 11.38 

processor-to-residency-time ratio for swappable TSO or 
batch work 11.69 

queue-length measurement II.38-II.39 

SIO condition code Is, indicating possible control unit 
contention II.42-II.43 

time to resolve page-in 11.61 

used to judge reported values of measurements II.5 

(see also rules-of-thumb) 
TCAM 

achieving consistent response time III.31 

blocksize of checkpoint data set III.30 

buffer hints III.29-III.30 

not run in APG III. 15 

queueing hints III.3 1 

real storage guidelines III.9 

utilization of fixed storage III.30 
TCB processor service units 

not recorded by RMF II. 1 3 
TCB time 

computing for SVCs II.21-II.23 

computing from RMF II.13-II.14 

example of TCB times for SVCs II.24-IL27 

focusing on 11.11 

from SMF 1.42 

used to compute working set size II.57-II.58 

identifying heavy users of II.29-II.31 

in relation to other categories of processor time 11.10 

not reported by SIR 11.10 

not reported by SMF 11.10 

not reported by RMF 11.10 

possible inaccuracy if processor stopped II. 1 1 
telecommunications access method 

(see TCAM) 
temporary data sets 

using VIO to reduce EXCP counts 11.47 
terminal wait swaps, reducing 

input 11.53 

output 11.53 
terminals, type of 

affecting TCAM buffer size III.29 
termination SMF record, job 

(see type 5 SMF record) 
think time as factor in TSO transaction rate 

computing 1.8 
thresholds for SRM indicators of system contention 

(see happy values) 
throughput 

measuring for batch 1.9 

possible degradation in shared DASD systems III.25 

setting throughput limits 1.12 

unsatisfactory 

(see system-oriented performance problems, analyzing) 
tools 

(see measurements tools) 
TPUT SVC with HOLD 

causing output te^- ^inal wait swaps 11.53 
trace activity from RMF 

using to monitor performance 1.46 
trace table analysis I.35-I.40 
example of I.39-I.40 
format of trace table entries I.35-I.37 
location of trace table 1.35 
notes on writing sampling program 1.40 
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trace table analysis (continued) 

overhead 1.35 

suggested uses 1. 38-1.39 

to identify SVCs called by specific jobs/logons 11.29 

to obtain processor cost of SVCs II.21-II.23 
track groups, number of 

(see &NUMTGV) 
TRANSACT macro instruction 

specifying scheduling priority 11.65 
transaction 

(see TSO transaction, IMS transactions) 
transaction response report 

measurements of IMS response time 1.8 
transaction, TSO 

(see TSO transaction) 
transaction rate, TSO 

(see TSO transaction rate) 
TSO environment 

(see TSO work) 
TSO MCP macro instruction 

specifying TCAM buffer size 111.29 
TSO response time 

achieving consistency via TCAM features III.31 

affected by number of user catalogs III.33 

affected by TCAM queueing III. 31 

analyzing 

(see user-oriented performance problems, analyzing) 

factor in formula for transaction rate 1.8 

reported by RMF or MF/1 1.7 

factors not included in measurement 1.8 
TSO transaction 

as defined by SRM I.7-I.8 

factors to include in documenting workload 1,10 
TSO transaction rate 

defined 1.8 

formula for 1.8 

reported by RMF or MF/1 1.8 
TSO work 

access to processor, investigating as possible delay 11.69 

comparison of ASM-queue-length and UIC happy values in 
IMS/batch and TSO/batch systems III.14 

controlling in IMS-mixed environment to reduce IMS 
paging III. 13 

demand paging target values 11.52 

dispatching priority hints III. 15 

enqueue contention, investigating as possible delay 11.75 

factors to include in documenting workload I.IO 

MPL hints for TSO domains III.17 

possible impact of shared DASD III.25 

real storage guidelines III.9 

reducing EXCPs for 11.47 

TCAM queueing III.31 
(see also TCAM) 

waiting for I/O, investigating as possible delay 11.7 3 

waiting to start, investigating as possible delay II.66-II.67 

2305 vs. additional real storage Ill.l 1 
TS-step termination SMF record 

(see type 34 SMF record) 
TSU 

(see &TSU) 
TTIMER 

reducing path length III.27 
tuning tools 

(see measurement tools) 
turnaround time, batch 

measuring 1.9 

analyzing 

(see user-oriented performance problems, analyzing) 



type 4 SMF record 

breakdown of demand paging by address space 11,57 

identifying programs with high TCB time 11.29 

measuring processor time 1.42 

measuring storage usage I.42-I.43 

source of reclaim rates 11.57 

used to compute working set size II.57-II.58 

used to investigate excessive swapping 11.77 

used to investigate possible problem of waiting for I/O 11.73 

used to investigate VIO paging 11.59 

used to obtain EXCP counts 11.47 
type 5 SMF record 

analyzing for excessive wait-to-start time for batch 11.65 

data on batch turnaround and throughput 1.9 

used to focus on heavy processor users 11.29 

used to investigate if work is delayed before being swapped 
back in 11.77 

used to investigate possible problem of work waiting for the 
processor 11.69 

used to measure processor time 1.42 
type 14 SMF record 

used to measure I/O activity 1.43 

used to obtain EXCP counts III.3 
type 15 SMF record 

used to measure I/O activity 1.43 

used to obtain EXCP counts III.3 
type 34 SMF record 

breakdown of demand paging by address space 11.57 

measuring processor time 1.42 

measuring storage usage 1.4 2-1.4 3 

source of reclaim rates 11.57 

used to compute working set size 11,57-11,58 

used to investigate excessive swapping 11,77 

used to investigate possible problem of waiting for 
I/O 11.73 

used to investigate VIO paging 11.59 

used to obtain EXCP counts 11.47 
type 35 SMF record 

used to focus on heavy processor users 11.29 

used to investigate if work is delayed before being swapped 
back in 11.77 

used to investigate possible problem of work waiting for the 
processor 11.69 

used to measure processor time 1.42 
type 70-76 SMF records 

measuring system-wide resource activity 1.43 
typing time as factor in TSO transaction rate 

computing 1,8 

UCB definitions 

reducing fixed storage 11,56 
UIC (unreferenced interval count) 

changing happy range to reduce IMS paging III. 14 

changing happy range to reduce multiprogramming 
level 11.58 

updating in SU 7 SRM, possible effect on IMS paging III. 13 
unilateral swaps, reducing 11.54 
unit record devices 

blocksize and buffer hints III.3 
UNITSIZE parameter on TSOMCP macro instruction 

specifying TCAM buffer size III.29 
unreferenced interval count 

(see UIC) 
user catalog 

(see VSAM catalogs) 
user control block, SRM (OUCB) 

as source of performance data 1.33-1,34 
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user extension block, SRM (OUXB) 

as source of performance data 1.34 
user-oriented performance objectives 
as basis of problem description II. 4 
batch turnaround time 1.9 
definition 1.6 

dictating focus of analysis II.6, II.8 
failure to meet 

(see user-oriented performance problems, analyzing) 
IMS response time 1.8 
noting discrepancies in measurements of I.l 1 
possible conflict with system-oriented objectives 1.6 
relationship to workload management II.6 
TSO response time 1.7 -1.8 
user-oriented performance problems, analyzing 
access to the processor, investigating possibility 
of II.69-II.70 
data base segment contention, investigating possibility 

of 11.75 
enqueue contention, investigating possibility of 11.75 
excessive demand paging, investigating possibility 
of 11.71 -11.72 
excessive swapping, investigating possibility of 11.77-11.78 
focus on workload management II.6, II. 8 
general approach II.63-II.64 
major causes of delay 11.63 
order to investigate causes of delay 11.64 
possibility of conflicting solutions 11.64 
switching focus to resource management 11.64 
waiting for I/O, investigating possibility of 
IMS 11.73 
TSO/batch 11.73 
waiting to start, investigating possibility of II.65-II.67 
batch 11.65 
IMS II.65-II.66 
TSO II.66-II.67 
user time as factor in TSO transaction rate 

computing 1.8 
users that monopolize system resources 

isee dominant users) 
usual values for measurements 

identifying II. 5 
utilities, IMS 

(see IMS DC Monitor; IMS/VS log transaction analysis 
utility program; IMS/VS program isolation trace report 
utility program; IMS/VS statistical analysis utility 
program) 
utilization of channels 

(see channels, investigating) 
utilization of data sets 

consideration when reducing I/O bottlenecks 11.45 

validity checking 

bypassing to reduce SVC path lengths III.27 
VIO 

used to reduce EXCP counts 11.47 
VIO loop 

resulting in auxiliary storage shortage swap 11.53 
VIO paging 

computing non-TCB time for VIO and demand 
paging II.15-II.19 
formula 11.18 
focusing on 

judging rates 11.49 

if paging costing non-productive processor time II.5 1 
if paging I/O interfering with other I/O 11.49-11.5 1 
investigating 11.59 



virtual storage 

control blocks that contain information on 1.2 8-1.29 

insufficient assigned to IMS work pools 11.65 

tools that report on I.28-I.29 

usage, measured by SMF I.42-I.43 
virtual storage analysis tool, IMS/VS 

general information 1.19 
virtual telecommunications access method 

(see VTAM) 
volume mounting 

operational bottleneck 1.49 

possible problem in shared DASD III. 25 
volume table of contents 

(see VTOC) 
volumes 

tools that report on 1.29 
VSAM catalogs 

buffer space III.33 

GDGs in VSAM catalogs III.33 

loading via DEFINE III.33 

multiple user catalogs 111.33 

on non-shared device III.33 

secondary allocation III.33 
VSAM data bases 

ways to reduce data base SIOs 11.48 
VTAM 

real storage guidelines III.9 
VTAM buffer trace analysis tool 

(see GTFVTAM) 
VTOC 

listings, as source of data 1.30 

placement of III.23 

using GTF to examine 1.44 
V=Rjobs 

resulting in requested swaps 11.53 

WAIT (SVC 1) 

reduced path length when executed by authorized 
program III.27 
wait swaps 

(see detected wait swaps; long wait swaps) 
wait time, processor 

(see processor, wait time) 
waiting for I/O to complete 

(see I/O, waiting for) 
waiting to start 

possible cause of user-oriented problem 11.63 
investigation and possible solutions II.65-II.67 
batch work 11.65 
IMS work II.65-II.66 
TSO work II.66-II.67 
weights, domain 

(see domain weights) 
working sets 

defined 11.57 

obtaining from measurement tools II.57-II.58 
RMF 11.57 
SIR 11.57 
SMF II.57-II.58 
separating users with large working sets 

as step in reducing demand paging 11.55 
ways to reduce 11.58 
workload activity report, MF/1 

measurement of TSO response time 1.7 -1.8 
measurement of TSO transaction rate 1.8 
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workload activity report, RMF 2305 devices 

measurement of TSO response time 1.7-1,8 vs. additional real storage III.l 1 

measurement of TSO transaction rate 1.8 

used to compute EXCP/SIO rate 11.47 3270s 

used to compute residency time for TSO trivial and tapes, configuration of III. 11 

transactions 11.66 3340-F device 

used to compute TCB time 11,13-11.14 VTOC placement on III.23 

used to compute wait-to-start time for TSO trivial 3350 device 

transactions 11.66 VTOC placement on III.23 

used to compute working set size II.57-II.58 

used to investigate possibility of excessive swapping 11.77 

used to monitor performance 1.45 
workload balancing 

(see workload management) 
workload, documenting L9-I.10 

measuring resource requirements I.11-I.12 
workload management 

avoiding conflicts between meeting user-oriented and system- 
oriented objectives 1.6 

documenting workload I.9-I.10 

focus of analysis for user-oriented problems II.6, II.8 

identifying heavy users of the processor II.29-II.31 

possible solution to data base segment contention 11.75 

possible solution to enqueue contention 11.75 

to reduce contention for I/O resources 11.35,11.45 

to reduce criticalness of processor II.9 

to separate users with large working sets 11.58 
work pools, IMS 

insufficient virtual storage assigned 11.65 
WTOR reply delays 

operational bottleneck 1.49 

XDAP 

(see EXCP/XDAP) 

zapping system 

as solution to performance problem II. 8 

&BUFSIZE 

JES2 hints III.19 
&DEBUG JES2 parameter, hint III.20 
&MAXJOBS 

possible effect of &NUMTGV on III. 1 9 

specifying III.21 
&NUMDA JES2 parameter, hint III.20 
&NUMTGV parameter 

performance implications of III. 19 

suggested values 

for different combinations of devices III. 20 

for different devices III. 19 
&STC 

subparameters to specify to reduce overhead 111,20 

&TSU 

subparameters to specify to reduce overhead 111,20 
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You can use this form to submit performance hints for 
possible publication in this book. It is understood that 
IBM and its affiliated companies shall have the non- 
exclusive right, in their discretion, to use, copy, and 
distribute all submitted information or material, in any 
form, for any and all purposes, without any obligation 
to the submitter, and that the submitter has the un- 
qualified right to submit such information or material 
upon such basis. When submitting performance hints, 
indicate the system and release level of your system and 
the SUs installed on your system 



PERFORMANCE 
NOTEBOOK 
INPUT 
FORM 

Your views about this publication might help improve its 
usefulness; this form will be sent to the author's depart- 
ment for appropriate action. Using this form to request 
system assistance or additional publications will delay 
response, however. For more direct handling of such 
requests, please contact your IBM representative or the 
IBM Branch Office serving your locality. 



What is your occupation? 

Number of latest Technical Newsletter (if any) concerning this publication: 
Please indicate your name, address, and phone number in the space below. 



IBM representative 



State problem and suggested solution or suggestions for analyzing problem. Attach examples, etc. , when available. 



Thank you for your cooperation. No postage stamp necessary if mailed in the U.S.A. 
(Elsewhere, an IBM office or representative will be happy to forward your comments.) 
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Your comments and performance hints, please . . . 

This manual is part of a library that serves as a reference source for system analysts, 
programmers, and operators of IBM systems. Your comments on the other side of this 
form will be carefully reviewed by the persons responsible for writing and publishing 
this material. All comments and suggestions become the property of IBM. 
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